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ILTROIUCTTCN 


irc Vis:cr. Hashes are informal vorkinp papers interded primarily 
tc stimulate inierral irterecticr amone participants in the A.I. 
I ate rat cry 'r Vi si or arc Rctotics proup. Mary of them report 
inifi-.iy tentative conclusions cr incomplete- work. Others deal 
v.'iti n:fhly detailed accounts cf local equipment and preprams 
that lack peroral interest. Still ethers are of preat 
importance, tut ls.cl the polish a.nd elatcrate attention to proper 
referencing that characterizes the more formal literature. 


nevertheless, tie Vision Flashes collectively represent the only 
documentation cf an important fraction cf the work dene in 
machine vision and rctotics. The xurpese of this report is to 


rake 


ti e findings more readily available, but since they are net 


revised as presented here, readers should keep in mind the 
cripinal purpose of the papers! 


many report on details of the K.I.T. blocks world vision system. 
The entire spectrum of vision processing is represented, frem lew 
level j.( atu.re iindnnp to hiph level scene analysis reouirirp 
extorsive vorlc knowledge ard deductive pewer. Cr. all levels, 
tney reflect a movement from ac hoc preprar s arc testirp toward 
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the scurd theorv thet ere expects from ruccesfull science. 


The root recent papers shift otter tier away from the now well 
ur.deretc od plane polyhren world and toward two new foci: 
i) real world vision 


r 

C 


:} applications for machines with virion. Careful study 


v;ilj shcv/ that wort in these new areas is productively puided ty 
the evclvinr ideas and metaphors of artificial intelligence in 
general and "by cur earlier work in simple visual worlds. 
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ABSTRACT 


This is an introduction to some of the f'.I.T. 
the last few years* The toxics discussed are 
line drawing semantics 2) heterarchy J) the 
Business and 4) copyiny scenes. All toxics a 
detail elsewhere in vision flashes or theses. 


A.I. vision work of 
1)t/altz's work or 
arciert learn:ny 
re discussed ir more 




his thesis was oriyinally published as Vision Flash 70. 
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irRCDUCTICN 


Rerearch ir machine 


vision is an important activity in 


artificial intelligence laboratories for tvo rajor reasons: 
First, understanding vision is a worthy subject for its own sal-re. 
The point cf view cf artificial intelligence allows a fresh new 


lock at old questions and exposes a great deal about vision in 
genera], independent of whether ran cr machine is the seeing 
agent. Second, the same problems found in understanding vision 
are of central interest in the development of a broad theory of 
intelligence. Making a machine see brings one to grips with 
problems like that of knowledge interaction on many levels and of 
large system organization. In vision these key issues are 
exibited with enough substance to be nontrivial and enough 
simplicity to be tractable. 

These objectives have led vision research at MIT tc focus 
on two particular goals: learning from examples ard copying from 
spare parts. Both goals are framed in terms cf a world of 
bricks, wedges, and ether simple shapes like those found in 
children's toy boxes. 

Good purposeful description is often fundamental to 
research in artificial intelligence, and learning how to do 
description constitutes a major part of our effort in vision 
research. This essay begins with a discussion of that part of 
scene analysis known as body finding. The intention is tc show 
how our understanding has evolved away from blind fumbling toward 
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sulstartive theory. 


The next section re\elves mound the ores nizational 
metaphors and the rules of good prepraminirp practice appropriate 
for thinkirp about, large knowledfe-oriented systems. Tindina 
proups of objects and using the groups to pet at the properties 
of their members illustrates concretely how some of the ideas 


about systems work out in detail. 

The topic of learning follows. Discussing learning is 
especially appropriate here not only because it is an important 
piece of artificial intelligence theory but also because it 
illustrates a particular use for the elaborate analysis machinery 
dealt with-in the previous sections. 
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EVOLUTION CF A SEMANTIC THFCRY 
Guzman and the Eody Protier 

The tody finding story bey ins with an ad hoc but crisp 
syntactic theory and ends in a simple, appealing theory with 
serious semantic roots. In this the history of the body finding 
problem, seems paradigmatic of vision system progress in general. 

Aldolfc Guzman started the work in this area (Guzman 
19C8)* I review his program here in order to anchor the 
discussion and show how better proprams emerge through the 
interaction of observation, experiment, and theory. 

The ta.sk is simply to partition the observed regions of a 
scene into distinct bodies. In figure 1, for example, a 
reasonable program would report something like (A F C) and (D E) 
as a plausible partitioning of the five regions into, in this 
case, two bodies* Keep in mind that the program is after only 
one good, believable answer. Many simple scenes have several 
equally justifiable interpreticns. 

Guzman's program operates on scenes in two distinct 
passes, both of which are quite straightforward. The first pass 
gathers local evidence and the second weighs that evidence and 
offers an opinion about how the regions should be grouped 
together into bodies* 

The local evidence pass uses the vertices to generate 
little pieces of evidence indicating which of the surrounding 
regions belong to the same body. These quanta of evidence are 
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called lirks. Figure 2 
chews hew each contributes 
aluavs argue that the she 


lists each vertex type recognised and 
to the set of links. The arrow lirks 
ft—bordering repiors belong together, 


the fork acre ambitiously provides three such links, ore for each 
pair of surrounding repions, ard so on. The resulting lirks for 
the scene in figure 1 are displayed superimposed on the origiral 


drewiny in figure la 


Internally the lirks are represented in 


list structure equivalent to the abstract diagram in fipure 3 b. 
There the circles each represent the corresrondinslv lettered 


region from fipure 3a. The arcs joining the. circles represent 
lirks. 


The job of pass two is to combine the link evidence into 
a parsiny hypothesis. How Guzman's pass two approached its final 
form nay he understood by imagining a little series of theories 
about how to use the eviderce to best advantage. Figure 3a is so 


c: 

K. /’ 


imp-le that almost any method will do. Consequently fipure 4 and 


firure 


are used to further illustrate the experimental 


observations behind the evolving* sequerce of theories. 

The first theory to think about is very simple. It 


argues that ary two repions belong to the same body if there is 
a link between them. The theory works fine on many scenes, 
certainly on those in figure 3a and fipure 4. It is easy, 
however, to think of examples that fool this theory because it is 
far toe inclined toward enthusiastic region binding. Wherever a 
coincidence produces ar accidental link, as for example the links 


















an error occurs in 


placed by the spurious rsi vertex in ft pure E, 
the direction of too much conglomeration. 

The problem is corrected in theory two. Theory two 
differs from theory one because it requires two links for binding 
rather than just one. By insisting on more eviderce, local 
evidence anomalies are diluted in their potential to d ame re the 
end result. Such a rethod works fine for figure 5, but as a 
general solution the two link scheme also faIters, now on the 
side of stinginess. In figure 4, partitioning ty this second 
theory yields (A B) (C) (D) (E F). 

This stinginess can also be fixed. The first step is to 
refine theory two into theory three by iterating the amalgamation 
procedure. The idea is to think of previously joined together 
region groups as subject themselves to conglomeration in the same 
way regions are joined. After one pass over the links of figure 
4, we have A and F joined together. But the combination is 
linked to C by two links, causing C to be sucked in on a second 

run through the linking loop. Theory three then produces (ABC) 
(D) (E F) as its opinion. 

Theory four supplements three by adding a simple special- 
case heuristic. If a region has only a single link to another 
region, they are combired. This brings figure 4 around to (A E C 
D) (E F) as the result, without re-introducing the generosity 
problem: that came up in figure 5 when using theory one. That 
scene is now also correctly separated into bodies. 
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Cnly one irore refinement is necessary tc complete this 
secuence of imagined theories and bring us close tc Guzman's 
firal program. The required addition is motivated by the scenes 
like that of figure 6. There v/e have agair toe much linkirg as a 
result of the indicatec 1 fork vertex. Although not really wrong, 
the ore object answer seems less likely to humans than a report 
of two objects. Guzman overcame this sort of problem toward the 
end of his thesis work not by augmenting still further the 
evidence weighing but rather by refining the way evidence is 
originally generated. The basic change is that all placement of 
links is subject tc inhibition by contrary evidence from adjacent 
vertices. In particular, no link is placed across a line if its 
other end is the barb of an arrow, a leg of an 1, or a part of 
the crossbar of a T. This is enough to correctly handle the 
problem of figure 6. Adding this link inhibition idea gives us 
Guzman's program in its final form. In the first pass the 
program gathers evidence through the vertex inspired links that 
are not inhibited by adjacent vertices. In the second pass, 
these links cause binding together whenever two regions or sets 
of previously bound regions are connected by two or more links. 
It is a somewhat complex but reasonably talented program which 
usually returns the most likely partition of a scene into bodies. 

But does this program of Guzman's constitute a theory? 
If we use an informal definition which associates the idea of 
useful theory with the idea of description, then certainly 
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Guzman's work is a theory of the repior parsinf aspect of vision, 
either as describee’ here or manifested in Guzman's actual machine 
propram. I must hasten to say, however, that it stards 
incomplete on some of the dimensions along which the worth of a 
theory can be measured. Guzman's proprsm was insiphtful and 
decisive to future developments, but as he left it, the theory 
had little of the deep semantic roots that a pood theory should 
have. 

Let us ask some questions to better understand why the 
program works instead of just how it works. . When does it do 
well? Why? When does it stumble? How can it be improved? 

Experimentation with the propram confirms that it works 
best or scenes composed of objects lackinp holes (Winston 1971) 
and havinp trihedral vertices. (A vertex is trihedral when 
exactly three faces of the object meet in three-dimensional space 
at that vertex.) 

Why should this be the case? The ansv.er is simply that 
trihedral vertices most often project into a line drawinp as L's, 
v/hich we ipnore,. and arrows and forks, which create links. The 
program succeeds whenever the weak reverse implicatior that 
arrows and forks come from trihedral vertices happens to be 
correct. Using the psi vertex amounts to a corollary which is 
necessary because we humans often stack things up and bury atn 
arrow-fork pair in the resulting alignment. From this point of 
view, the Guzman program becomes a one-heuristic theory ir which 






a. 


link is created whenever a picture vertex rap have ccrae from a 
trihedral apace vertex. 

hut when doer the heuristic fail? Apein experiments 
provide sorething cf ar answer. The trihedral vertex heuristic 
most often fails when alignment creates yerjurous arrows. 
Without seme sort of link inhibition mechanism, it is easy to 
construct examples littered with bad arrows. Tc combat pcor 
evidence, two possibilities must be explored. Cne is to demand 
more evidence, and the other is to find tetter evidence. The 
complexity and much cf the arbitrary quality of Guzman's work 

results from electing to use more evidence. But using more 

* 

evidence was not enough. Guzman was still forced to improve the 
evidence via the link inhibition heuristic. 

The startling fact discovered by Eugene Freuder is that 
lirk inhibition is enough! With some slight extensions to the 
Guzman inhibition heuristics (Eattner 1970), complicated evidence 
weighing is unnecessary. A program that binds with one link does 
about as well as mere involved ones. By going into the semantic 
justification for the generation of links, we have a better 
understanding of the body linking problem and we have a better, 
more adequate program to replace the original one. This was a 
serious step in the right direction. 

Shadows 

Continuing to tra.ee the development of MIT's scene 
understanding programs, the next topic is a sertie irto the 
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T 


question of handling shadows. The first work at MIT on the 
subject was done by Crban (Crhar 1°70). Fis purpose was to 


eliminate or erase shadows from a drawing. The approach was 
ouite Guzman—3ike in flavor as Crban worked empirically with 

.4. -1 ■ V 

vertices, tryirg to learn their language and discover heuristic 
clues that would help establish shadow hypotheses. He found that 
ouite complex scenes could be handled through the following 


simple facts: 1) a shadow boundary often displays two or more L 

type vertices in a row 2) shadow boundaries tend to form psi type 

vertices when they intersect a straight line and 3) shadows may 

often he found by way of the L's and followed through psi's. 

* 

Crban's program is objectionable in the same way Guzman's 
is. Namely, it is largely empirical and lacking ir firm semantic 
roots. The ideas work in some complex scenes orly to fail in 
others. Particularly troublesome is the common situation where 


short shadow boundaries involve no L type vertices. 

After Crban's program, the shadow problem remaired at 
pasture for some time. The issue was avoided by placing the 
light source near the eye, thus eliminating the problem by 
eliminating the shadows. Aside from being disgusting 
aesthetically, this is a poor solution because shadows should be 
a positive help rather than a hindrance to be erased out and 
forgotten. 

Interest in shadows was reawakened in conjunction with a 
desire to use more knowledge of the three-dimensional world in 
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scene analysis. Among the obvious 

1) The world of blocks 


facts are the following: 
and wedges has a preponderance 


cf vertical lines. Given that a scene has a single 
distant light source, these vertical lines all cast 


shadows at the Sc me angle or the retina. Fence when 


cne line 


is identified as a shadow, it renders all 


ether lines at the saire angle suspect. 

2) Vertical lines cast vertical shadows or vertical 
fares. 


2) Horizontal lines cast shadows on horizontal faces 
that are parallel to the shadow casting edges. 

4) If a shadow line emerges from a vertex, that vertex 
almost certainly touches the shadow bearing surface. 

With these facts, it is ea.sv to thirk about a program 

that would crawl through the scene of figure 7, associating 

shadow boundaries with their parent edges as shown. One could 
van implement something, through poirt four, that would allow 
the system to knew that the cube in figure 7 is lying on the 

table rather than floating above it. Such a set of programs 

would be on the same level as Freuder's refinement cf Guzman's 


program with respect to semantic flavor. V;e were in fact on the 
verge of implementing such a program when Waltz radicalized cur 
understanding cf beth the shadow work and the old body-finding 
problem. 
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Figure 7 

Siirple heuristics allow 


edpes causing them. 


*■‘#9?^. ilS^' 


shadow lines to be associated with the 






Waltz and £emartic Interpretation 
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This section deals with the enormously successful vork of 
1/aJtz (haltz 1972a) (Waltz 197fb). Readers familiar with either 
the v/ork cf huff ran (Huffman 1971) or that of Clowes (Cloves 
1971) will instantly recognize that their work 
considerable foundation on which Waltz's theory rests. 


ie 


the 


A line in a drawing appears because of one or another of 
several possibilities in the physical structure: The line may be 
a shadow, it may be a crack between two aliyred objects, it may 


be the seam, between two surfaces we see, or it may be the 
boundary between an object and whatever is in back of it. 


It is easy encuyh to label all the lines in a drawing 

according to their particular cause ir the physical world. The 

drawing in figure £, for example, shows the Huffman labels for a 

» 

cube lying flat or the table. The plus labels represent seams 
where the observer sees both surfaces and stands on the convex 
side cf the surfaces with the inside of the object lying on the 
corcave. The minus labels indicate the observer is cn the 
concave side. And the arrowed lines indicate a boundary where 
the observer sees only one of the surfaces that form the physical 


A curious and amazing thing about such labeled line 
drawings is that only a few of the ccmbiratorially possible 
arengererts cf labels around a vertex are physically possible. 
We will never see a L type vertex with both wings labeled plus no 




Figure 8 

Huffman latles for a cute. Plus implies a convex edge, minus 
implies concave, and an arrow implies only one of the edge- 
forming surfaces is*visible. 
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ratter how many legal Dine drawings we exarire. (It is presur ed 
that the objects are huilt of trihedral vertices and that the 
viewpoint is such that certain types of coincidental alignment in 
the picture denair arc lacking.) Indeed it is easy to prove that 
an enureration of all possibilities allowed by three-dimersioral 
constraints includes only six possible I vertex labelirgs and 
three each of the fork ard arrow types. These are shewn in 
figure 9* 


Given the constraints the world nieces 


or 


the 


arrangements of line labels around a vertex, ore can sc the other 


way. Instead of usirp knowledge of the real physical structure 
to assign semantic labels, one car use the known constraints on 
how a drawing can possibly be labeled to get at an understanding 
of what the physical structure must be like. 


The vertices of a line drawing are like the pieces of a 
jigsaw puzzle in that both are limited as to how they can fit 
together. Selections for adjacent vertex labelings simply cannot 
reouire different labels fer the line between them. Giver this 
fact a simple search scheme car work through a drawing, assigning 
labels to vertices as it gees, taking care that no vertex 
labeling is assigned that is incompatible with a previous 
selection at an adjacent vertex. If the search fails without 
finding a compatible set of labels, then the drawing cannot 
represent a real structure. If it does find a set of DabeDs, 
then the successful set or sets of labels vield much information 
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about the structure. 

Waltz generalized the basic ideas in two fundamental 
wags. First he expanded the set of line labels such that each 
includes much more information about the physical situation. 
Second, he devised a filtering procedure that converges cn the 
possible interpretations with lightning speed relative to a mere 
obvious depth-first search strategy. 

Waltz's labels carry information beth about the cause of 
the line and about the illumination on the twe adjacent regions. 
Figure 10 gives Waltz's eleven allowed line interpretations. The 
set includes shadows and cracks. The regions beside the line are 
considered to be either illuminated, shadowed by facing away from 
the light, or shadowed by another object.Thesepossibilities 
suggest that the set cf legal labels would include 11 X J X J = 
99 entries, but a few simple facts immediately eliminate about 
half cf these. A concave edge ray rot, for example, have one 
constituent surface illuminated and the other shadowed. 

With this set cf labels, bedy finding is easy! The line 
labels with arrows as part of their symbol (two, three, four, 
five, nine, ten, and eleven) indicate places where one body 
obscures another body or the table. Once Waltz's program finds a 
compatible set of line labels for a drawing, each body is 
surrounded by line labels from the arrew class. 

To create his program, Waltz first worked cut what vertex 
configurations are possible with his set of line labels* Figure 
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1 / CONVEX EDGE 


» 




OBSCURING EDGES — OBSCURING BODY LIES TO 
3 ) RIGHT OF ARROW'S DIRECTION 










CRACKS — OBSCURING BODY LIES TO RIGHT OF 
ARROW'S DIRECTION 


SHADOWS — ARROWS POINT TO SHADOWED REGION 


8 


CONCAVE EDGE 


> 


SEPARABLE CONCAVE EDGES — OBSCURING BODY 


- e — 

10 > 

LiLJ Oium v r nru\uw 

DOUBLE ARROW INDICATES 

-*— 

• n' 

MEET ALONG THE LINE 


Figure 10 

Lire interpretations recognized by Waltz's program* 
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APPROXIMATE NUMBER 

APPROXIMATE NUMBER 


OF COMBINATORIALLY 

OF PHYSICALLY 


POSSIBLE LABELINGS 

POSSIBLE LABELINGS 


2,500 

80 


125,000 

70 

~~K 

125,000 

500 

V 

125,000 

# 

500 

yf> 

6 x 10 6 

10 


6 x 10 6 

300 


6 x 10 6 

100 

X 

6 x 10 6 

100 

)< 

6 x 10 6 

100 


3 x 10 8 

30 


Only a few of the combinatcrially possible labelings are 
physically possible. 



Page 29 


11 fives the result. Happily the possible vertex labelings 
ccrstitute only a tinv fractior of the wavs labels can be arraved 

V t/ Ks ft 

around a vertex. The number of possible vertices is large but 

not unrans.fea.blv so. 

Increasing the nur ber of legal vertex labelings dees not 
increase the number of interpretations of typical line drawings. 
This is because a proper increase in descriptive detail strongly 
constrains the way things may go topethen. Again the analogy 
with pip saw puzzles fives an idea of what is happ>eninf: The 
shapes of pieces constrain how they may fit together, but the 
colors give still more constraint by adding arother dimension of 
comparison. 

Interestingly, the number of ways to label a fork is much 
larger than the number for an arrow. A single arrow ccnseouently 

offers more constraint and less ambiguity than does a fork. This 

explains why experiments with Guzman's program shewed arrows to 

K 

be more reliable than forks as sources of good links. 

Figure 12 shows a fairly complex scene. Put with little 
effort, Waltz's program can sort cut the shadow lines and find 
the correct number of bodies. 

V/hat I have discussed of this theory' so far is but an 
hors d'oeuvre. Waltz's forthcoming doctoral dissertation has 
much to say about handling coincidental alignment, finding the 
approximate orientation of surfaces, and dealing with higher 
order object relations like support (Waltz 1972b). Fut without 
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petting into those exciting results, I can ccmmert cn hew his 
work fits together with previous ideas cn hcdy finding and on 
shadows. 

First of all Waltz's program has a syntactic flavor. The 
program has a table of possible vertices and or some level can be 
thought to parse the scene. Eut it is essential to understand 
that this is a program with substantive semantic roots. The 
table is net an amalgam: of the purely ad hoc and empirical. It 
is derived directly from arguments about how real structures can 
project onto a two dimensional drawing. The resulting label set, 
together with the program that uses it, can be thought of quite 
well as a compiled form of those arguments whereby facts about 
three-dimensional space become constraints on lines and vertices. 

In retrospect, I see Waltz's work as the culmination of a 
long effort beginning with Guzman and moving through the work of 
Orban, Ratrer, Winston, Huffman and Clowes. Each person built on 
earlier ideas and experiments, producing either a refinement, a 
reaction, or an explanation. The net result is a tradition 
moving toward more and better ability to describe and toward more 
and better theoretical justification behind working programs. 




SYSTEM ISSUES 
Heterarchy 

Waltz's work js pert cf understanding how lire drawings 
corvey information about scenes. This section discusses some of 
our newer ideas about how to pet such understanding into a 
working system. 

At MIT the first success in copying a simple block 
structure from spare parts involved using a pass-oriented 
structure like that illustrated ir figure 13. The solid lines 
represent data flow and the dashed lines, control. The executive 
in this approach is a very simple sequence of subroutine calls, 
mostly partitioned into one module. The calling up of the action 
modules is fixed in advance and the crder is indifferent to the 
peculiarities cf the scene. Each action module is charged with 

augmenting the data it receives according to its labeled 
specialty. 

This kind of organization does not wcrk well. Its 
purpose is to make a vehicle quickly fcr testing the modules then 
available. It is often better to have one system working before 
expending too much effort in arguing about which system is best. 

From this base we have mcved toward another style of 
organization which has come to be called heterarchical (Minsky 
and Parert 1972). The concept lacks precise definition, but the 
following ere some of the characteristics that we aim for: 

A complex system should be gcal oriented. 


1 . 
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Procedures at all levels should he short and associated 
vith sore definite goal. Goals should normally he 
satisfied hy invoking a small number of subgoals for 
ether procedures or hy directly calling a few 
primitives*. A corollary is that the system should be 
top down. For the most part nothing should be done 
unless necessary to accomplish something at a higher 
level. 

2. The executive control should he distributed 
throughout the system. In a heterarchical system, the 
modules interact not like a master and slaves but more 
like a community of experts. 

Programmers should make as few assumptions as 
possible about the state in which the system will be 
when a procedure is called. The procedure itself 

should contain the necessary machinery to set up 
whatever conditions are required before it can do its 
job. This is obviously of prime importance when many 
authors contribute to the system, for they should be 
able to add knowledge via new code without completely 
understanding the rest of the system. In practice this 
usually works out as a list of goals lying like a 
preamble near the beginning of a routine. Typically 
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& ■ 4 


these goals ere satisfied by simple refererce to the 
data bs.se, hut if net, rotes are left as to where helo 

^ -i. 

re.y he found, in the PLAIIIFF? (Hewitt 1979) or CCNNIVER 
rtyle (McDermott and Sussman 1979). 

A- The system should contain sore knowledge of itself. 
It is not enough to think of executives and primitives. 
There shoule be nodules that act as critics and 
complain v/hen something looks suspicious. Others must 
know how and when the primitives are likely to fail. 
Communication among these modules should be more 
colorful than mere flow of data and command. It should 
include what in human discourse would be called advice, 
suggestions, remarks, complaints, criticism, questions, 
answers, lies, and conjectures. 

9. A system should have facilities for tentative 
conclusions. The system will detect mistakes as it 
goes. A conjectured configuration ray be found to be 
unstable or the hand may be led to grasp air. V/hen 
this happens, we need to know what facts in the data 
base are most problematical; we need to know how to 
^**7 fix things; and we need to know how far—ranging 
the consequences of a change are likely to be. 





Graphically such a system loohs mere like a retwerk of 
procedures than an orderly, immutable sequence. Each procedure 
is corrected to others via potential control transfer links. In 
practice which of these links is used depends on the context in 
which the various procedures are used, the context being the 
joint product of the system and the problem undergoing analysis. 

Note particularly that this arrangement forces us to 
refine our concept of higher versus lower level routines. Now 
programs normally thought to be low level may very well employ 
other programs considered high level. The terms no longer 
indicate the order in which a routine occurs in analysis. 
Instead a vision system procedure is high or low level according 
to the sort of data it works with. Line finders that work with 
intensity points are low level but may certainly on occasion call 
a stability tester that works with relatively high level object 
models. 

Firin and Environment Priven Analysis 

Cur earliest MIT vision system interacted only narrowly 
and in a predetermined way with its environment. The pass 
oriented structure prevents better interaction. Fut we are now 
moving toward a different sort of vision system in which the 
environment controls the analysis. (This idea was prominent in 
Ernst's very early work (Ernst 1961).) 

Readers who find this idea strange should see an 
exposition of the notion by Simon (Simon 1969). 


He argues that 



much of whet passes as intelligent behavior is in point of fact a 
harry cooperation between unexpectedly simple algorithms and 
complex environments. He cites the case of an ant wardering 
along a leach rift with ant sized obstacles. The ant's 
curvacious path might seem to be an insanely complex ritual to 
someone locking only at a history of it traced on paper. But in 

fact the humble ant is merely trying to circumvert the beach's 
obstacles and go home. 

Watching the Iccus of control of our current system as it 
struggles with a complicated scene is like watching Simon's ant. 
The up and down, the around and backing off, the use of this 
method then another, all seem to be mysterious at first. Eut 
like the ant's, the system's complex behavior is the product of 
simple algorithms coupled together and driver by the demands of 
the scene. The remainder of this section discusses seme element 
procedures implemented by Finin which illustrate two ways in 
which the environment influences the MIT vision system (Finin 
1972). 

The vision system contains a specialist whose task is to 
determine what we call the skeleton of a brick. A skeleton 
consists of a set of three lines, one lying along each of the 
three axes (Finin 1972). Fach of the lines in a skeleton must be 
complete and unobscured so that the dimensions of the brick in 
question may be determined. Figure 14 shows some of the 
skeletons found in various situations by this module. 








The only problem with the program lien in the fact that 
complete skeletons are moderately rare in practice because of 
heavy obscuring Ever in the simple arch ir figure 15a, one 
object, the left side support, cannot be fully analyzed, lacking 
as it does a completely exposed lire ir the depth dimersior. Fut 
humans have no trouble circumventing this difficulty. Indeed, it 
generally does not even occur to us that there is a problem 
because we so naturally assume that the right and left supports 
have the same dimensions. At this point let us look at the 
system's internal discourse when working on this scene to better 
understand how a group - hypothesize - criticize cycle typically 
works out: 

Let me see, what are A's dimensions. First I must identify 
a skeleton. Cops! V/e can only get a partial skeleton, two 

complete lines are there, but only a partial line along the 
third brick axis. This means I know two dimensions but I 
have only a lower bound on the third. Let me see if A is 
part of some group. Oh yes, A and B both support C so they 
form a group of a sort. Let me therefore hypothesize that A 
and B are the same and run through my check list to see if 
there is any reason to doubt that. 

Are A and B the same sort of objects? 

Yes, Both are bricks. 

Are they both oriented the same way? 



m&- 


Figure 15 
In one cas 
hypothesis 
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Yes, that checks out toe. 

Well, dc the observeble dimensions iratch? 

Indeed. 

Is there any reasor to believe the unobservable 

dimension cf A is different from its enalcgue on 
' E? 

No. 

OK. Everything seems all right. j will 

tentatively accept the hypothesis and proceed. 

Through this irternal dialogue, the machine succeeds in 

finding all the necessary dimensions for the cbscured support in 

figure 1^,a. Figure 15b shows how the conflict search can fail at 
the very last step. 

Grouping amounts, cf course, to using a great deal of 
context in scene analysis. We have discussed how the system uses 
groups to hypothesize properties for the group's members and we 
should add that the formation of a greup is in itself a matter 
hypothesis followed by a search for evidence conflicting with the 
hypothesis. The system now forms group hypotheses from the 
following configurations, roughly in order of grouping strength: 

1. Stacks or rows of objects connected by chains of 
support cr in-front-of relations. 

2. Objects that serve the same furcticn such as the 
sides of an arch or the legs of a table. 
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7. Objects that are close together. 

4. Objects that are of the same type. 

To test the validity of these hypotheses, the irachine 
makes tests of good membership on the individual elements. It 
basica31y performs corformity tests* throwing out arything too 
unusual. There is a preliminary theory of hov this can be done 

sensibly (Winston 1971)* The basic feature cf Winston's theory 
is that it involves not only a measure cf how distant a 
particular element is from the norm, but also of how much 
deviation from the norm is typical and thus acceptable. 

Note that this hypethesis-rooted theory is much different 
from Gestaltist notions of good groups emerging magically from 
the set cf all possible groups. Critics cf artificial 
intelligence correctly point out the computational implausibility 
of considering all possible groups but semehew fail to see the 
alternative of using clues to hypothesize a limited number of 
good candidate groups. 

Naturally all of these group - hypothesize - criticize 
efforts are less likely to work out than are programs which 
operate through direct observation* It is therefore good to 
leave data base notes relating facts both to their degree of 
certainty and to the programs that found them. Thus an assertion 
that says a particular brick has such and such a size may well 
have other assertions describing it as only probable, conjectured 
from the dimensions of a related brick, and owing the discovered 
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realticnship to a particular grouping* program. Using such 
1ncwledge is as yet orly rlanred, but in preparation we try to 
refrair from using mere than one method in a single program. 
This makes it easy to describe hew a particular assertion was 
made by simply noting the name of the program that made it. 

Visual observation of movement provides another vay the 
environment can influence and control what a visior system thinks 
about* One of the first successful projects was executed at 
Stanford (Vdckman 1967). The purpose was tc align two bricks, 
one atep the other* The method essentially required the complete 
construction of a line drawing with subsequent determination of 

relative position. The Japanese have used a similar approach in 
placing a block inside a box. 


The MIT entry into this area is a little different. V/e 
do not recuire complete recomputation of a scene, as did the 
Stanford system* The problem is to check the position of a just 
placed object to be sure it lies within some tclerance of the 
assigned place for it. (In our arm errors in placement may 
occasionally be on the order of 1/2".) 

Rather than recompute a line drawing of the scene to find 
the object s coordinates, we use our model cf where the object 
should be to direct the eye to selected key regions. In brief, 
what happens is as follows: 

1* The three-dimensional coordinates for selected 
vertices are determined for the object whose position 
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is tc be checked. 

2. Ther the supposed locations of those vertices on 
the eye's retina are easily computed. 

/' vertex search using circular scans around each of 
these supposed vertex positions hill climbs to a set of 
actual coordinates for the vertices on the retina 
(Winston and Lerman 1972). Revised three—dimersioral 
coordinates can be determined from these retinal 
coordinates, given the altitude of the object. 

4* Comparing the object's real and supposed 
coordinates gives a correction which is then effected 
by a gentle, wrist-dominated arm action. 

The vertex-locating program tries to avoid vertices that 
form alignments with those of other objects already in place. 
This considerably simplifies the work of the vertex finder. With 
a bit more work, the program could be made to avoid vertices 
obscured by the hand, thus allowing performance of the feedback 
operation more dynamically, without withdrawing the hard. 
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IE/RNirc TC IDFI'TIFY TCY ETCCK STRUCTUFFS 
learnirp 


This section describes a worh inp comm ter program which 

embodies a new theory cf learning (Winston 197C). I believe it 

is uni ike previous theories because its Tasic idea is to 

understand how concepts can be learned from a few judiciously 

selected examples. The sequence in Figure 16, fcr exararfie 

generates in the machire an idea of the arch sufficient to handle 

correctly all the configurations in figure 17 in spite of severe 

rotations, size changes, proportion changes ard changes in 
viewinp anple. 

Althouph no previous theory in the artificial 

intelligence, psych.olopy, or other literatures can completely 

account fcr anything like this competence, the basic ideas are 
quite simple: 


1. If you want to teach a concept, you must first be 
sure your student, man or machine, can build 
descriptions adequate to represent that concept. 

2. If you want to teach a ccncept, you should use 
samples which are a kind of ron—examrle. 

The first point on description should be clear. At some 
level we must have an adequate set of primitive concepts and 
relations cut of which we can assemble interesting concepts at 
the next higher level which in turn become the primitives for 
concepts at a still higher level. The operation cf the learning 





ARCH 
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program depends completely on the power of the analysis programs 
described in the previous sections. 

But what is meant by the second claim that one must show 
the machine net just examples of concepts but something else? 
First of all, something else means something which is close to 
being an example but fails to be admissable by way of ore or a 
few crucial deficiencies. I call these samples near-misses. My 
view is that they are more important to learning than examples 
and they provide just the right information to teach the machine 
directly, via a few samples, rather than, laboriously and 
uncertainly through many samples in some kind of reinforcement 
mode. 

The purpose of this learning process is to create in the 
machine whatever is needed to identify instances of learned 
concepts. This leads directly to the notion of a model. To be 
precise, I use the term as follows: 

A model is a proper description augmented by 
information about which elements of the description are 
essential and by information about what, if anything, 
must not be present in examples cf the concept. 

The description must be a proper description because the 
descriptive language — the possible relations — must naturally 
be appropriate to the definitions expected. For this reason one 
cannot build a model on top cf a data base that describes the 
scene in terms of only vertex coordinates, for such a description 
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is on too lew a level. For can one build a model or tor of a 

higher level description that cortairs only color information, 

for example, because that information is usually irrelevant to 
the concept in question. 

The key part of the definition of model is the idea that 
some elements of the description must be underlined as 
particularly important. Figure 18 shows a training sequence that 
conveys the idea of the pedestal. The first step is to show the 
machine a sample cf the concept to be learned. From a line 
drawing, the scene analysis routines produce a hierarchical 
symbolic description which carries the same sort of information 
about a scene that a human uses and understands. Flocks are 
described as bricks cr wedges, as standing or lying, and as 
related to others by relations like in-frort-of or supports. 

This description resides in the data base in the form of 

list structures,, but T present it here as a network of nodes and 

pointers, the nodes representing objects and the pointers 

representing relations between them. See figure 19 where a 

pedestal network is shown. In this case, there are relatively 

few things in the net: just a node representing the scene as a 

whole and two more for the objects. These are related to each 

other by the supported-by pointer and to the general knowledge of 

the net via pointers like is-a, denoting set membership, and has- 

posture, which leads if* one case to standing and in the other to 
lying. 
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Nov; in the pedestal, the support relation is essential — 
there is no pedestal without it. Similarly the posture and 
identity of the hoard and "brick must he correct. Therefore, the 
objective in a teaching sequence is to somehow convey to the 
machine the essential, emphatic quality of those features (Later 
on we will see further examples where some relations become less 
essential and ethers are forbidden). 

Returning to figure 18 note that the second sample is a 
near-miss in which nothing has changed except that the board no 
longer rests on the standing brick. This is reflected in the 
description by the absence of a supperted-by pointer. It is a 
simple matter for a description comparison program to detect this 
missing relation as the only difference between this description 
and the original one which was an admissable instance. The 
machine can only conclude, as we would, that the loss of this 
relation explains why the near-miss fails to qualify as a 
pedestal. This being the case, the proper action is clear. The 
machine makes a note that the supported-by relation is essential 
by replacing the original pointer with must-be-supported-by. 
Again note that this point is conveyed directly by a single 
drawing, not by a statistical inference from a boring hoard of 
trials. Note further that this information is quite high level. 
It will be discerned in scenes as long as the descriptive 
routines have the power to analyze that scene. Thus we need not 

be as concerned about the simple changes that incapacitate older. 
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lover level learning ideas. Potations, size dilations ard the 

like ere easily handled, river the descriptive power we have in 
operating programs. 

Continuing now with our example, the teacher proceeds to 
basically strengthen the other relations according to whatever 
prejudices he has* In this sequence the teacher has chosen to 
reinforce the pointers which determine that the support is 
standing and the pointers which similarly determine that the 
supported object is a lying board. Figure 20 shows the model 
resulting. 

Now that the basic idea is clear, the slightly more 
complex arch sequence will bring out some further points. The 
first sample, shown back in Figure 16 is an example, as always. 
From it we generate an initial description as before. The next 
step is similar to the one taken with the pedestal in that the 
teacher presents a near—miss with the supported object now 
removed and resting on the table. But this time net one, but two 
differences are noticed in the corresponding description networks 
as now there are two missing supported—by pointers. 

This opens up the big question of what is to be done when 
more than one relationship can explain why the near-miss misses. 
What is needed , of coursej is a theory of how to sort cut 
observed differences sc that the most impertart and most likely 
to be responsible difference can be hypothesized and reacted to. 

The theory itself is somewhat detailed, but it is the 
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PART-IS 


PART-IS 




MUST-BE-SUPPORTED-BY 


MUST-HAVE-POSTURE 


MUST-BE-A 


IS-A 


MUST-HAVE-POSTURE 


STANDING 


LYING 




BOARD 


BRICK 


__ •••-■■: v 

, Figure 20 ' VUXX:: 
4C pedestal model. 







exflcrrtior of this detail through writing* snd 6x*nfriTnentir^ with 
programs that gives the overall theory a. crisp substance. 
Repeated cycles of refinement and testing of a theory, as 

embodied in a program, is an important part cf an erer^ina 
artificial intelligence methodology. 

Nov/ the results of this approach on the difference 
ranking module itself include the following points: 

First of all, if two differences are observed which are 
of the same nature and description, then they are assumed to 
contribute jointly to the failure of the near—miss and both axe 
acted cn. This handles the arch case where two surport relations 
were observed to be absent in the near-miss. Since the 
differences are both of the missirg pointer type and since both 
involve the same suppcrted-by relation, it is deemed 
heuristically sound to handle them both together as a unit. 

Secondly, differences are ranked in order of their 
distance from the origin of the net. Thus a difference observed 
in the relationship of two objects is considered more important 
than a change in the shape of an object's face, which in turn is 
interpreted as more important than an obscured vertex. 

Thirdly, differences at the same level axe ranked 
according to type* In the current implementation, differences of 
the missing pointer type are ranked ahead of those where a 
pointer is added in the rear—miss.* This is reasonable since 
droppirg a pointer to make a rear-miss may well force the 
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introduction of a new pointer. Indeed we have ignored the 
introduction of a support pointer between the lying brick end the 
table because the difference resulting from this new poirter is 
inferior to the difference resulting from the missing pointer. 
Finally, if two differences ere found of the same type on the 
same level, then some secondary heuristics are used to try to 
sort them out* Support relations, for example, make mere 
important differences than one expects from touch or left-right 
pointers. 

Now these factors constitute only a theory of hypothesis 
formation- The theory dees make mistakes, especially if the 
teacher is poor* I will return to this problem after completing 
the tour through the arch example. Recall that the machine 
learned the importance of the support relations. In the next 
step it learns, somewhat indirectly,, about the hcle. This is 
conveyed through the near-miss with the two side supports 
touchirp. How the theorjr of most important differences reports 
that tv/o new touch pointers are present in the near-miss, 
symmetrically indicating that the side supports have moved 
together. Here surely the reasonable conclusion is that the new 
pointers have fouled the concept. The model is therefore refined 
to have must«-*not~touch pointers between the nodes of the side 
supports- This dissuades identification pregrams, later 
described, from ever reporting an arch if such a forbidden 
relation is in fact present. 
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It is row clear hov crucial information of the negative 
sort is introduced into models. They can contain net only 
inf erection about v hs,t is essential hut also information about 
what sorts of characteristics prevent a sample from being 
associated with the modeled concept. 

So far I have shown examples of emphatic relations, both 
of the must-be and must-rot-be type as introduced by near-miss 
samples. The following is an example of the inductive 
generalization introduced by the sample with the lying brick 
replaced by a wedge. Whether to call this a kind of arch or 
report it as a near-miss depends on the taste of the machine's 
instructor, of course. Let us explore the consequence of 
introducing it as an example, rather than a near—miss. 

In terms of the description network comparison, the 
machine finds an is-a pointer moved over from brick to wedge. 
There are, given this observation, a variety of things to do. 
The simplest is to take the most conservative stance and form a 
nev. class, that of the brick or wedge, a kind of superset. 

To see what other options are available, look in figure 
21 at the descriptions of brick and wedge and the portion of the 
general knowledge net that relates them together. There various 
sets are linked together by the a-kind-cf relationship. From 
this diagram we see that our first choice was a conservative 
point on a spectrum whose ether end suggests that we move the is- 
a pointer over to object, object being the most distant 












Page 59 

intersection cf a.—kind—of relations. V/e choose a conservative 
position and fix the is—a pointer to the closest observed 
intersection, in this case right-prism. 

Again a hypothesis has to he Ea.de, and the hypothesis may 
well he wrong. In this case it is a cuestion of difference 
interpretation rather than the cuestion of sorting out the 
correct difference from many, hut the effect is the same* There 

simply must he mechanisms for detecting errors and correcting 
them. 

Errors are detected when an example refutes a previously 
mace assumption. If the first scene cf Figure 22 is reported as 
an example of concept X while the second is given as a near-miss, 
the natural interpretation is that an X must he standing. But an 
alternate interpretation, considered secondary by the ranking 
program, is that an X must not be lying. If a shrewd teacher 
wishes to force the secondary interpretation, he need only give 
the tilted brick as an example, for it has no standing pointer 
and thus is a contradiction to the primary hypothesis. Under 
these conditions, the system is prepared to hack up and try an 
alternative. As the alternative may also lead to trouble, the 
process of backup may iterate as a pure depth first search. Cne 
could do better by devising a little theory that would hack up 

more intelligently to the decision most likely tb have caused the 
error. 

I mentioned just now the role of a shrewd teacher. I 
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Figure 22 

A training sequence that leads to backup 
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refard the dependence on s tescher ar a feature cf this theory. 
Toe often in the pest history cf mechire learning theory the use 
of a teacher was considered cheating and mechanisms were instead 
expected tc self—organize their way tc understanding by way of 
evolutionary trial ard error, or reinforcement, or whatever. 

ignores the very real fact that humans as well as machines 
learn very little without gcod teaching. The first attempt 
should be to understand the kind of learning that is at once the 
most common and the most useful. 

It is clear that the system assimilates new models from 
the teacher and it is in fact dependent on good teaching, hut it 
depends fundamentally on its own good judgement and previously 
learned ideas to understand and disentangle what the teacher has 
in mind* It must itself deduce what are the salient ideas in the 
training sequence and it roust itself decide on an augmentation of 
the model which captures those ideas. By carefully limiting the 
teacher to the presentation of a sequence of samples, low level 
rote learning questions are avoided while allowing study of the 

issues which urderly all sorts of meaningful learning, including 
interesting forms of direct telling. 

Identification 

Having developed the theory of learning models, I shall 
say a little about using them in identification. Since this 
subject is both tangential to the main thrust and documented 
elsewhere (Winston 1970), I shall merely give the highlights 
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here. 

To begin with, identification is dore in a variety of 
nodes, our system already exhibiting the following three: 

1. We may present a scene and ask the system to 
identify it. 

2. We may present a scene with several concepts 
represented and ask the system to identify all of them. 

We may ask if a giver scene contains an instance of 
something. 

Of course, the first mode of identifying a whole scene is 
the easiest. V/e simply insist that 1) all models must-be-type 
pointers are present in the scene's description and 2) all the 
models must-not-be-type pointers must rot be present. Tor 
further refinement, we look at all other differences between the 
model and scene of other than the emphatic variety and judge the 
firmness of model identification according to their number and 
type. 

«V X 

When a scene contains many identifiable rows, stacks, or 
other groups, we must modify the identification program to allow 
for the possibility that essential relations may be missing 
because of obscuring objects* The properties of rows and stacks 
terd to propagate from the most observable member unless there is 
contrary evidence* 

The last task, that of searching a scere for a particular 
concept is a wide open question. The method now is to simply 




feed cur retwerk matching preprarc both the model and the larper 
network and hope for the "best. If some objects are matched 
■ geinst corresponding parts of the model, their pointers to other 
extraneous objects are forpotten, and the identification routine 
is applied. huch remains to be dene slonp the lines of fuidinm 
the match contextually to the ripht pa.rt of the scene. 
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COPYING TOY BLOCK STRUCTURES 

I here give a "brief description cf the system's higher 
level functions along vith a scenario giving their interaction in 
a very simple situation. The main purpose is to illustrate the 
top down, goal oriented and environment dependent flavor of the 
system. Cede samples are available elsewhere (V/inston 1971) 

Figure 23 shows the possible call paths between some of 
the programs. Note in particular the network quality that 
distinguishes the system from the earlier pass oriented metaphor. 

Clarity requires that only a portion of the system be 
described. In particular, the diagram and the discussion omits 


the following: 

1) A large number of antecedant and erasing programs 
which keep the blocks world model up to date. 

2) A large network of programs which find skeletons and 
locate lines with particular characteristics. 

7) A large network of programs that uses the group — 
hypothesize - criticize idea tc find otherwise 
inaccessible properties of hidden objects. 

4) A network of programs that jiggles an object if the 
arm errs too much when placing it. 

The Functions 


COPY 

As Figure 23 shows, CCPY simply activates programs that 

namely, it calls for 


handle the two phases of a. copying problem; 







FIND-STORAGE MOVE ,/ / FIND-PART CHOOSE-TO-PLACE 



Figure 23 

The virion system. 
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the errre parts tc be fourd and rut away into the spare parts 
warehouse area, and it initiates the replication cf the rew 
scene. 

STOFE-P/RTS 

To disassemble a scene and store it, STOFE-P/RTS loops 
through a series cf operations. It calls appropriate routines 
for selecting an object, finding a place for it, and for enacting 
the movement tc storage. 


CHCOSE—TO—FEMOVE 

The first body examined by CFOCFF—TO-REMOVE comes 
directly from a successful effort to amalgamate some regions into 
a body using FIND-NFW-FCDY. /fter some body is created, CHCOSE- 
TC-REMOVE uses FIND-FFLON to mahe sure it is not underneath 
something. Frequently, some cf the regions surrounding a newly 
found body are not yet cornected to bodies, so FIND-BELOV has a 
request link tc BIRD-REGION. (The bodies so fourd, of course, 
are placed in the data base and are later selected by CHOOSE-TO- 
REFOVE without appeal to FIND-FEW—BCDY.) 

EIND-NEW-ECDY 

FIND-NEW-BCDY locates some unattached region and sets 
BIND-REGION to work on it. BIND-REGICN then calls collection of 


programs by Eugene Freuder which do a local parse and make 
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assertions cf the form: 

(R17 IE-A-FACF-CF F2) 

(E2 IS-A BODY) 

These programs appeal to a complicated retwcrk of subroutines 
that drive line find, inf and vertex find inf primitives arourd the 
scene looking for complete reficns (Winston 1972). 

FIND-BEIOW 

As mentioned, some refions may need parsinf before it 
makes sense to ask if a fiven object is below .something. After 
assurinf itself that an adjacent refion is attached to a body, 
FIHD-BFLOW calls the FIND—ABCVE proframs to do the work of 
determining if the body originally in question lies below the 
object owning that adjacent region. 

FIND-AEOVF-1 and FIND-ABOVF-2 and FIND-ABOVE-? 

The heuristics implemented in Winston's thesis (Winston 
1970) and many of these only proposed there are now working in 
the FIND-AFOVE programs. They naturally have a collection of 
subordinate programs and a link to BIND-REGION for use in the 
rvent an unbodied region is encountered. The assertions made are 
of the form: 

(B3 IS-ABOVE B7) 

HOVE 

To move an object tc its spare parts position, the 
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locations and dimensions are gathered up. Then MANIPULATE 
interfaces to the machine language programs driving the arm. 
After MOVE succeeds, STOKE—PARTS makes an assertion of the form: 

(B12 IS-A SPAREPART) 


FIND-TOP 

The first task in making the location calculations is to 
identify line-drawing coordinates of a "block's top. Then FIND- 
TALLNESS and FIND-ALTITUDE supply other information needed to 
properly supply the routine that transforms line-drawing 
coordinates to X Y Z coordinates. Resulting assertions are; 

(B1 HAS-DIMENSICNS (2.2 3.1 1.7)) 

(B1 IS-AT (47.0 -17.0 5.2 .3)) 

Where the number lists are of the form: 


(< smaller x—y plane dimension > 

< larger > 

<tallness>) 

(< x coordinate > <y> <z> <angle>) 

The x y z coordinates are those of the center of the bottom of 


the brick and the angle is that of the long x~y plane axis of the 
.brick with respect to the x axis. Two auxiliary programs make 
.the following assertions wherever appropriate: 


(B12 HAS-POSTURE 


( STANDING) 


LYING 
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(B7 IS-A 


(CUBE ^ 
BRICK 
STICK 
BOARD 





FIND-DIMENSIONS 

This program uses FIND-TOP to get the information 
necessary to convert drawing coordinates to three-dimensional 
coordinates. If the top is totally obscured, then it appeals 
instead to FIND-BOTTOM and FIND-TALLNESS-2. 

SKELETON 

SKELETON identifies connected sets of 3 lines which 

define the dimensions of a brick (Finin 1971) (Finin 1972). It 

and the programs under it are frequently called to find instances 
of various types of lines. 

FIND-TALLNESS-1 

Determining the tallness of a brick requires observation 
of a complete vertical line belonging to it. FIND-TALLNESS-1 
uses some of SKELETON'S repertoire of subroutines to find a good 
vertical. To convert from two-dimensional to three-dimensional 
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coordinates, the altitude of the hrick must also he known. 

FIND-TALLNESS-2 

Another program for tallness looks upward rather than 
downward. It assumes the altitude of a block can be found but no 
complete vertical line is present which would give the t alln ess. 
It tries to find the altitude of a block above the one in 
question by touching it with the hand. Subtracting the altitude 
of the lower block from that of the higher gives the desired 
tallness. 


FIND-ALTITUDE 

FIND-ALTITUDE determines the height of an object's base 
primarily by finding its supporting object or objects. If 
necessary, it will use the arm to try to touch the object's top 
and then subtract its tallness. 

FIND-SUPPORTS 

This subroutine uses FIND-SUPPORT-CANDIDATES to collect 
together those objects that may possibly be supports. FIND- 
SUPPORT-CANDIDATES decides that a candidate is in fact a support 
if its top is known to be as high as that of any other support 
candidate. If the height of a candidate's top is unknown but a 
lower bound on that height equals the height of known supports, 
then ADD—TO—SUPPORTS judges it also to be a valid support. At 
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the moment the system has no understanding of gravity. 

FIND-STORAGE 

Once an object is chosen for removal, FIND-STORAGE checks 
the warehouse area for an appropriate place to put it. 

MAKF-COPY 

To make the copy, MAKE-COPY, CHCOSE-TO-PLACF, and FIND- 
PART replace STORE-PARTS, CHOOSF-TO-REMOVE and FIND-STORAGE. 
Assertions of the form: 

(B12 IS-A SPAREPART) 

(B2 IS-A-PART-CF COPY) 

(B2 IS-ABOVE El) 

are kept up to date throughout by appropriate routines. 

CHCOSE-TO-PLACF 

Objects are placed after it is insured that their 
supports are already placed. 

FIND-PART 

The part to be used from the warehouse is selected so as 
to minimize the difference in dimensions of the matched objects. 
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A Scenerio 

In what follows the scere in figure 24a provides the 
spare parts which first must he put away in the warehouse. The 
scene to he copied is that of Figure 24h. 

COPY 

COPY begins the activities. 

STORE-PARTS 

STORE-PARTS begins supervision of disassembly.. 

CH OOSE-TO-REMOVE 

FIND-NEW-EODY 

BIND-REGION 

CHOOSE-TO-REMOVE parses a few regions together into a 
body, B1. A great deal of work goes into finding these regions 
by intelligent driving of low level line and vertex finding 
primitives. 

FIND-BELOW 

BIND-REGION 

FIND-ABOVE 

A check is made to insure that the body is not below 
anything. Note that B2 is parsed during this phase as required 
for the FIND-ABOVE routines. Unfortunately B1 is below B2 and 
therefore CHOOSE-TO-REMOVE must select an alternative for 
removal. 

FIND-BELOW 

FIND-ABOVE 

B2 was found while checking out B1* CHCOSE-TO-REMOVE now 
notices it in the data base and confirms that it is not below 
anything. 
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FIND-STORAGE 


FIND-STORAGE finds an empty spot in the warehouse. 
MOVE 


MOVE initiates the work of finding the location and 
dimensions of B2. 

FIND-TOP 

FIND-ALTITUDE 
FIND-SUPPORTS 

FIND-SUPPORT-CANDIDATES 

FIND-TOP-HEIGHT 

FIND-ALTITUDE 

FIND-SUPPORTS 

FIND-SUPPORT-CANDIDATES 

FIND-TOP-HEIGHT 

FIND-TAILNESS-1 

FIND-TALLNESS-1 

FIND-BOTTOM proceeds to nail down location parameters for 
B2. As indicated by the depth of call, this requires something 
of a detour as one must first know B2's altitude, which in turn 
requires some facts about B1. Note that no calls are made to 
FIND-ABOVE routines during this sequence as those programs 
previously were used on both B1 and B2 in determining their 
suitability for removal. 

FIND-DIMENSIONS 

A call to FIND-DIMENSIONS succeeds immediately as the 
necessary facts for finding dimensions were already found in the 
. course of finding location. Routines establish that B2 is a 
lying brick. 


MANIPULATE 

MANIPULATE executes the necessary motion. 
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CHOCSE-TO-REMOVE 

FIND-EELOW 

FIND-STORAGE 

B2 is established as appropriate for transfer to the 
warehouse, A place is found for it there. 

MOVE 

FIND-TOP 

FIND-DIMENSIONS 

MANIPULATE 

The move goes off straightforwardly, as essential facts 
are in the data base as side effects of previous calculations. 

CHOCSE-TO-REMOVE 

FIND-NEW-EODY 

No more objects are located in the scene. At this point 

the scene to be copied, figure 24, is placed in front of the eye 

and analysis proceeds on it. 

MAKE-COPY 

CHOOSE-TO-PLACE 

FIND-NEW-BODY 

BIND-REGION 

B3 is found. 

FIND-BELOW 

BIND-REGION 

FIND-ABOVE 

B3 is established as ready to be copied with a spare 


part. 


FIND-PART 

FIND-DIMENSIONS 

FIND-TOP 


\ 


Before a part can be found, B3*s dimensions must be 
found. The first program, FIND-TOP, fails. 


FIND-BOTTOM 
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FIND-ALTITUDE 

FIND-SUPPORTS 

FIND-SUPPORT-C ANDIDATFS 
FIND-TOP-HEIGHT 

FIND—DIMENSIONS tries ar. alternative for calculating 

dimensions. It starts by finding the altitude of the bottom. 

FIND-TA1LNESS-2 
FIND-SUPPORTED 
FIND-BELOW 
FIND-ABOVE 
FIND-SUPPORTS 

FIND-SUPPCRT-CANDIDATES 

FIND-TAILNESS-2 discovers B4 is above B3. 

FIND-ALTITUDE 

TOUCH-TOP 

FIND-TAILNESS-1 

FIND-ALTITUDE finds B4's altitude by using the hand to 
touch its top subtracting its tallness. B3's height is found by 
subtracting B3*s altitude from that of B4. 

MOVE 

MANIPULATE 

Moving in a spare part for B3 is now easy. E3's location 

was found while dealing with its dimensions. 

CHOOSE-TO-PLACE 

FIND-BELOW 

FIND-PART 

FIND-DIMENSIONS 

FIND-TOP 

MOVE 

MANIPULATE 

Placing a part for B4 is easy as the essential facts are 

now already in the data base. 

CHOOSE-TO-REMOVE 

FIND-NEW-EODY 
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No other parts are found in the scene to be copied. 


Success. 
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cor cluting remarks 

This essay began with the claiir that the study of vision 
contributes both tc artificial intelligence and to a theory of 
vision. Working with a view toward these purposes has occupied 
many years of study at MIT and elsewhere on the toy world of 
simple polyhedra. The progress in semantic rooted scene 
analysis, learning, and copying have now brought us to a plateau 
where we expect to spend seme time deciding what the next 
important problems are and where to lock for solutions. 

The complete system, which occupies on-the order of 
10C,00C thirty-six bit words, is authored by direct contributions 
in code from over a dozen people. This essay has rot summarized, 
but rather has only hinted at the difficulty and complexity of 
the problems this group has faced. Many important issues have 
not been touched on here at all. Line finding, for example, is a 
task on which everything rests and has by itself occupied more 
effort than all the other work described here (Roberts 196?) 
(Kerskcvits and Binford 1970) (Griffith, 1970) (Horn 1971) (Shirai 
1972). 





Page 79 



References 




acres, K, "Cn £ Seeinr TMnrs," Artificial Intellijence, Vo3. 2, 

, H., Cor puter-Operated Mechanical Hand," D.fc. 

.tepartmert of Electrical Engineering. 

M.I.T., December, 1961. ’ 

Finn, T., Skeleton of a Frick,” Vision FUash 19, 

Artificial Intelligence Laboratory, M.I.T., August, 1971. 

Fir.in, T., "A Vision Potpourri,” Vision Flash 26, Artificial 

Intelligence Laboratory, M.I.T., June, 197?. 

Griffith, A.,_"Computer Recognition of Prismatic Sclids,” RAC 

Technical Report 77, Project MAC, M.I.T., August, 1970. 

Guzman, A*, ”Computer Recognition cf Three-Dimensicnal Objects in 

a Usual Scene,” MAC Technical Report 99, Project MAC 
M.I.T., December, 1968. J 

Herskovits, A. and Einford, T., ”0r Boundary 1 ’ Detection,” A.I. 

JuJLv* 1076 ^ r ^' ! Ucial Intelligence Laboratory, M.I.T., 

Hev/itt, C., ’’Description and Theoretical Aralysis (Using 

Schemata) of PLANNER: A Language for Provirg Theorems and 
Manipulating Mcdeis in a Robot,” A.I. Technical Report 
\cyj2 Artlflclal Intelligence Laboratory, M.I.T., April, 

Korn, E. K. P., "The Binfprd-Hcrn line Finder,” Vision Flash 16, 

Artificial Intelligence Laboratory, M.I.T., June, 1971. 

Huffman, D.., "Impossible Objects as Nonsense Sentences," Machine 

Intelligence 6, pp. 299-327 (ed. Meltzer, F. & Michie, 
D.J, Edinburgh University Press, Edinburgh, 1971. 

McDermott, D. and Sussman, C t , -The COOTIVH) Reference Manual," 

^*2^, ^ n ^ e ^i£ ence laboratory, 

Minsky, M. and Papert, S., "Progress Report,” A. I. Memo. 252, 

^“7_^ lcla ^ Intelligence Laboratory, M.I.T., January, 



Page 80 


Crban, R., "Removirg Shadows ir a Scene," /-.I. Memo. 192, 

Artificial Intelligence Latoratory, M.I.T., August, 1970. 

Shirai, Y.., "A Heterarchical Program fcr Recogriticn of 

Polyhedra," A.I. Memo. 263, Artificial Intelligence 
Latoratory, M.I.T., June, 1972. 

Simon, K.., The Sciences of the Artificial, M.I.T. Press, M.I.T., 

Cambridge, 1969« 

Sussmar, G., Winograd, T., and Charniah, E., "MICRO-PLANNER 

Reference Manuel," A.I. Memo. 203A, Artificial 
Intelligence Laboratory, M.I.T., December, 1971. 

Rattner, M., "Extending Guzman's SEE Program," A.I. Memo. 204, 

Artificial Intelligence Latoratory, M.I.T., July, 1970. 

Roberts, L., "Machine Percepticn of Three-Dimensional Solids," 

Technical Report 315, Lincoln laboratory, M.I.T., May, 
1963. 

Waltz, D., "Shedding Light on Shadows," Vision Plash 29, 

Artificial Intelligence Laboratory, M.I.T., July, 1972. 

Waltz, D., Doctoral Dissertation and A.I. Technical Rejort in 

preparation. Artificial Intelligence laboratory, M.I.T. 

Wichmar, W., "Use of Ortical Feedback in the Computer Control of 

an Arm," A.I. Memo. 56, Stanford Artificial Intelligence 
Project, Stanford University, August, 1967. 

Winston, P. K., "Holes," A.I. Memo. 163, Artificial Intelligence 

Laboratory, M.I.T., August, 1968. 

Winston, P. H., "Learning Structural Descriptions from Examples," 

A.I. Technical Report 231, Artificial Intelligence 
Laboratory, M.I.T., September, 1970. 

Winston, P. H., "Wandering About the Top of the Robot," Vision 

Flash 15, Artificial Intelligence Laboratory, M.I.T., 
June, 1971. 

Winston, P. H#, "Wizard," Vision Flash 24, in preparation. 

Artificial Intelligence Laboratory, M.I.T., 1972. 

Winston, P." H. and Lerman, J, "Circular Scan," Vision Flash 23, 

Artificial Intelligence Laboratory, M.I.T., March, 1972. 





Page 81 



SHEEDINC LICFT CN SHADOW'S 
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July 1972 


ABSTRACT 


Thie rarer describee methods which allow a program to analyze ard 
interpret a variety of scenes made up of polvhedra with trihedral 
vertices. Scenes may contain shadows, accidental edge 
alignments, and some missing lines. This v;crk is based on ideas 
proposed initially by Huffman ard Clowes; I have aded methods 
which enable the pregram to use a number of facts about the 
physical world to constrain the possible interpretations of a 
line drawing, and have also introduced a far richer set of 
descriptions than previous programs have used. 


This paper was originally published as Vision Plash 29. 
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1.C INTRODUCTION 


ow c’o we ascertain the shares cf urfai iliar objects? 


VI cc we so reldcr Corfu re shadows with real thirrs? Fov do 
^ ' 

v/c "factor out" shadows when loclinp at scenes? Fov are we 
rile to see the v/c rid as essentially the same whether it is a 
bright sunny day, an overcast day, or a niyht with only 
street lights for illumine tier? In the terms of this pater, 
hrv err v/e reco,anise the identity of figures 1.1 end 1.2? Do 
vo use learnirf and hrov/ledye to interpret what we see, or do 


vc 


somehow automaticaliv see the world as stable and 


irdependent of liphtinp? V/hat portions of scenes can v/e 
understand from local features alone, and v/hat confirmations 
reouire the use of flc-bal hvoctheses? 


Various theories have beer proposed to explain how 
people extract three-dimensional information from scenes 
(Citscn 1950 is an excellent reference). It is well krown 
that ve pet depth and distance information from motion 
parallax and, for objects fairly close tc us, from eye focus 
feedback and parallax. Put this does not explain hov v/e are 
able to understand the three-dimensional nature of 
rheterraphed scenes. Perhaps we acquire knowledge of the 
shapes cf objects by hand1inf them ard rovirf around them, 
ard use rote memory to ass if r. shape to those objects wher v/e 



























































Page 85 

CTCUr- 1.0 


reccyrize ther in sceres. Pvt tl is dees ret exrlrir how we 

a. 

err. perceive the shapes of objects we hrve never peer, before, 
ilrrly, ti e feet that we car toll 


C-? r 

IO' U. i ■ 


rrccrtrin 


share, theuyt 


tell 

the 

shares 

of TT-nv 

f 

ation as 

a lire 

dravinr 

or 

ether 

fine det 

ails tc 

cf 

cours 

■e use 

texture 


yrrdiorts and ether details tc define certain edres. 


7 urcertcck this research with the belief that it is 
resaille to discover rules with which a preyram car obtain a 
tl roe-dircnsicral model of a scere, river only a reasonably 
feed lino dravinr of a scere. Such a preyram miyht have 
apriications both ir. practical situations and in developinr 
better theories cf human vision. Introspectively, J do not 
icel that there is a proa t difference beta yen serins 
"reality” and. seeiny line drawings. 


I oreever, there are considerable difficulties loth in 
process inf stereo ins yes (such as the problem cf decidin'- 
which points on each retina correspond to the same scene 
rcir.t; see Cuzman 1968, Lcrrar 1970) and in buildirf a. 
syster incorpcratiry hand-eye cccrdinaticn which could be 
usee to help explore ard cisarbiyuate a scere (Caschniy 
1771). It seems tc re that while the use of ranye finders, 
multiple liyht sources tc help, eliminate shadows (Sliirai 
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1S71)> and the rertrietier of scenes to known objects way all 
prove useful for practical robots, these an proa oh es avoid 
ccr i rr to grins v;:‘ th the rrture of human rercerticr vis-a-vis 
tic : r.rlicit three—d.ir.ensicna 1 information in lire draw:nas 


rea 1 


scenes. Vlhile I would be very cautious a lout 


clsirirr parallels between the rules in mv ore era r ard hr nan 


visual processes,, at the very least I have demonstrated a 
number of capable vision programs v/hich require only fired, 
men ocular line- dra vines fer their operation. 


In this thesis I describe a. working collection of 
computer program s which reconstruct three-dimensional 
descriptions fror. line drawings v/h ich are obtained from 
scenes composed of plane-faced objects under various lighting 


conditions. In this description the system identifies shadow 
lines and regions, groups regions which belong tc the same 
object, and notices such relations as contact or lack of 


contact between the objects, support and in—front—of/behind 
relations between the objects as well as information about 
the spacial orientation cf various regions, all using the 
description it has generated. 



1.1 DFSCRIPTICKS 


1 he cvere.ll goal of the raster is to rrovice a rrecise 

*■' *■ . 5 .' 

description of a plausible scene which could five rise tc a, 
particular lire drawirg. It is therefore irrcrtart tc have a 


feed ]anguage in which to describe features of scenes. 


Since 


I wish to have the program operate on unfamiliar objects, the 


language rust be capable of describing such object 


c 


The 


larpuape I have used is an expansion cf the labelinp system 
dcvelcped by Huffman (Huffman 1971) in the Urited States and 
Clever (Clever 1971) in Great Fritain. 


The lanpuape employs labels 
sc arer ts and refions in tbe scene 
ec:re peoretry, the connection or 


which are assipred to line 
These labels describe the 
lack of correction between 


adjacent repiens, the orientation of each repicn ir three 
dimensions, and the nature of the illumire.ticr for each 
repior (illuminated, projected shadow repion, or region 
facinp away from the light source). The pool of the propram 
is to assign a single label value to each line and repicn in 
the line drawing, except in cases where humans alsc fird a 
feature tc be ambiguous. 


cf such 


n 


hi 




language allows precise defir:tiorr 


concepts es supported by, in front of, behind, rests against, 
shadcvs, is shadowed tv, is capable of supporting, leers or, 


arc others 


'bus, if it is possible to label each ferture of 


cj K 


■cone uniouely, then it is r.cssihle to cirectlv extract 


these relatiors from the description cf the scene based or 


th i 


rheD ins. 


1.1 JITCTICN I ABETS 


Such of the pro,man's power is based on access to lists 
of possible line label assignments for each type cf function 
ir a Dine drawing. While a natural language analogy to these 
labels could be misleading, I think that it helps in 
explaining the basic operation of this portion of the 
orogram. 

Ji. ^ 


If v.e think of each possible label for a lire as a* 
letter ir the alphabet, then each function rust be 
labeled with an ordered list cf ’’letters" to form a 
legal "word” in the language. Thus each "word" 
represents a physically possible interpretation for a 
given junction. Furthermore, each "word" must match the 
"words" for surrounding jurcticns in order to form a 
legal "phrase", and all "phrases" in the scere must 
agree to form a legal "sentence" for the entire scene. 
The knowledge of the system is contained in (1) a 
dictionary made up cf every legal "word" for each type 
cf junction, and (2) rules by which "words" can legally 
combine with other "words". The range of the dictionary 
entries defines the uriverse of the program; this 
universe can he expanded by adding new entries 
svstenaticallv tc the dictionary. 




t: r r jr - i 

c. -• v/. j V • j 


• r 




in feet, the "dictionary " need not "be a stored ljst. 


Tie c ic tic-nary can cons jet cf 


re lativel’ 


OJ;.d 


11 list of 


possible edpe peoi.etries for each junction tyre, and a set of 
rules which can he used tc generate the cornuete dictiorarv 
H er., the eng: nal lists#. Depending cr the 8mount of corrxvter 
memory avails tie, it my either be desirable to store the 
cor: r le to iists as compiled knowledge cr tc pererate the lists 
vl cr they are needed. Ir rr.y current preprar the lists are 


fer tl c most rart precompiled 


The 


,he composition cf the dictionary ir interesting in its 
c\r ripht.1 bile sore basic edpe pcometries pive rise tc 
rrrry dictionary entries, sore pive rise to very few. 
total number cf ertries sharirp the same edpe .pecretry car be 
as lev as three for sere ARF.CV; functions, irclucinr shadow 
eepes, while the number generated by some FCFK juretion edpe 
fecretries is over 27C,00C (ircludinp repion criertaticn 
illumination values). 
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Li CTICI; I.AEFI AS^IGNITNT 


There: 

is 

a corsiccrab! 

c ax cur 1 of 

[._» 

r> 

o 

oj 

1—i 

information 

which can 1 

be us 

ed tc 

select 

subset of t.h 

e total 

rur her of 

di c tic rar* 

1 

er t 

rics 

which arc 

consistent 

with a 

rarticular 

junction. 

The 

first 

piece cf 

information i. 

s rlrer 

d ir included 
* ■ 


ixrlicitly in the ider cf junctior tyre. Junction? ere turned 
according to the rumber of line? which make it the ‘unction 
end the two dir.ersicral arrargenert cf these lines. Figure 
1.7 shews e.ll the junction types which car occur in the 
universe cf the pregrar. The dictionary is arranged by 
junction tvre, and a standard orderinf is assigned to all the 

v.y V -i * >. \ 

line segments which make ur junctions (exceut FOF.KS and 
MTTTT). 


The program can also use local region, brightness and 
line segment direction to preclude the assignment of certain 
labels to lines. For example, if it knows that ore region is 
brighter thar an adjacent region, then the line which 
separates the regions can be labeled as a shadow region in 
only one way. There are other rules which relate region 
orientation, light placement and region illumination as veil 
as rules which limit the rumber cf labels which car be 
assigned to line segments which border the support, surface 
for the scene. The program, is able tc combine all these 
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t’ rep. of irformat:or jr firdirp a; Hot cf amrcorirte labels 

e i. <• 

for a sinfle function. 


I.d CCIXIlATia: RILFS 


Ccrbination ruler ere used to relect from tie initial 
assirrr.er ts ti e label, or lab els, which correctly c escribe 


the scene 

fea 

tore s 

that 

could have 

rroduced c 

A. 

"aoh junction in 

t’ e given 

1 i no 

dra 

wirf.. 

The si ml 

J. 

est tvre 

V a. 

of combination 

rule :orel 

y st 

a tea 

that a 

label is a 

. possible 

descrirtion for 


a junction if and cnlv if there is at least one label which 
"matches" it assigned to each adjacent function. Two 
iurcticn labels "retch" if anc on]v if the lire segment which 
joins the junctions gets the sane interpretation from both of 
the junctions at its ends, 


Cf course, each interpretation (line label) is really a. 
shorthand code for a number of properties of the line and its 
adjoin inf repions. If the program can show that any one of 

these constituent values carrot occur in the fiver scene 
content, then the whole complex of values for that line 
exrressed imrlicitly in the irtermetatior cannot be possible 
either and, furthermore, any junction label which assigns 
this interpretation to the line segment can be eliminated as 
well. Thus, when it chooses a label to describe a 







rvcizc: i./i 


particular juncticn, it constrains all the junctions which 
surround the regions touching this juncticn, even though the 
ccnbir at ion rules only compare adjacert junctions. 


l ore col plicated rules are reeded if it is recessarv to 


relate junctions which do not share a visible region or line 

segir.ert. For erarple, I thought at the outset of mv work 

' ' »_' 

trat it right be necessary to construct models of hidden 
vertices or features which faced away from the eve in order 


to find unicue labels for the visible ' features. 


The 


difficulty in this is that unless a program can find which 
lines rejresent obscuring edges, it cannot knew where to 
construct hidden features, but if it needs the hidden 
features to label the lines, it may not be able to decide 
which lines represent obscuring edges. As it turns cu.t, no 
such ccmrlicated rules end constructions are necessary in 
general; most of the labeling problem can be solved by a 
scheme which only compares adjacent junctions. 


r 

L 
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1.5 FhPERIRENTAL RESULTS 


hen I be gar to write a program to implement the system 
I had devised, I expected to use a tree search system to find, 
which labels cr "lords" ecuId be assigned to each Inaction. 
Fcwever, the number cf dictionary entries fer each tvre of 
junction is very high, (there are slmcst 7 COO differert wavs 
tc label a PORK junction before even considering the 'possible 


repi or orientations!j sc I decided tc 


use 


o 

c 


C 4 


•ert 


of 


If -f 

j. 


iltering program" before doing a full tiee search. 


The program computes the full list of dictionary entries 
for each junction ir: the scene, eliminates from the list 
these labels which can be precluded on the basis of local 
features, assigns each reduced list to its iurcticn, and then 
the filterinp program, computes the possible labels for each- 
line, using the fact that a line label is possible if and 
only if there is at least one junction label at each end of 
the line which contains the line label. Thus, the list of 
possible labels for a line segment- is the intersection of the 
two lists of possibilities computed from the junction labels 
at the ends of the line segment. If any junction label would 
assign a interpretation tc the line segment which is not in 
this intersection list, then that label can be eliminated 


from consideration 


The filtering program uses a network 




r:r; 




iteration 


s 


cheme to syslema.tica]ly rer ove afl the 


interpretations which are precluded by the elirirrticr of 
labels at a particular iurcticn. 


then I ran this filtering program I v;as amazed to find 
that in the first few scenes I tried, this propram fourd a 


uricue label for each line 


Ever when I tried consider?hiv 


sere cornyliested scenes, there were orly 


a few 3ines in 


perera1 which were not uniquely specified, and sore of these 
wore essentially ambiguous, i.e. I 'could not decide exactly 
what sort of cepe pave rise to the lire sepmert myself.. The 
other amiipuities, i.e. the ones which I cculc resclve 
myself, ir peneral require that the propram recopnize lines 
which are parallel or collinear or regions which meet along 
mere than one lire sepmert, ar.d hence require rcre pi rial 
apreement. 


1 have leen able tc use this system to investigate a 
larpe number cf line drawings, ircluding ones with missing 
lines and ones with numerous accidentally aligned jurcticns. 
Prom these investigations I CB XI SB y with some certainty which 
types of scene features can be handled by the filtering 
program and which require more complicated processing, 
blether or net mere processing is required, the filtering 


k...: 


stem provides 


computationally cheap method for 


acauinng 
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Cl 


yreat deal of information. Per example, in neat ccer.cs a 


larye percentage of the lire seymonts ere uramhiyuously 
la tele <3, end more complicated prccessiny car he directed to 
ti e areas which rere.ir amhiyucus. As another example, if I 
erly wish to know which liner are shadows or which lines are 
tic cuts ice ec res of cbiects cr hew many chiects there are in 

"* l.ts 


:ie scene, the preyrsm may he able to yet this inf creation 
ever thoryh some ambiyuities remain, since the artim/itv ir.av 
crly involve reyicn illumination type or reyicn crier tat ion. 


Ilyure 1.^ shews some of the sceres which. the prcyrair is 
able to handle. The segments which remain amhiyuors after 
its oreration are marked with stars, ard the approximate 
si: curt of tir e the rroyram requires to label each scene is 
marked below it. The computer is a LDP-1C, and the proyram 
is written partially in MICLO-PIANNIE (Sussran et al 1971) 


arc; partially in compiled LIST 
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1.6 CCKPALISOI VJI7F CThTE VITTOr TRCGtAK? 


T ^r C'trcr + p? 

* \ w »o Ov/4. 


ciffers 


frcr previously proposed ores in 


revere 1 ii. 


port art wavs: 


irst, : t ir pile tc handle a much "broader rarer of 


re ere tvres than 


have previous preyrars 


m 


ire prorram 


"i. 


rd err tare’s" shadows, acre junctions vhich have missing 


lircs, ard apparent aliynrort- of edres caused Try the 

t/ 

particular placement of the eye with respect to the scene, so 

that no special effort, needs to he made tc avoid problematic 
fortures. 


Record, the desiyn of the preyram facilitates its 


irteyraticn with line—findirs r roars ns 


c; 


no hirher-lovel 


propr?ms such a.s rrcyrams which deal with natural lanwuaye or 
cversj.l system ycsls. The system can he used to write a. 
preyram which automatically requests and uses many different 
tyros of information tc find the possible interpretations for 
a single feature or pertior. of a scene. 


Third, the program is able to deal with ambiyuity in a 


nr turr 1 ir.r nner . £ 


cr.e features in a scene can be ambiruous to 


c 


person - eck:ry at the same scene and the preyrar rreserves 


tl ere various possibilities.. 


This toierarce for ar-biruitv is 
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if a rinfie internrotation is required fcr sere 
car he chosen free this list Try heuristic rules. 


Fourth, the program is alporithmic end does not reouire 

-i. 

facilities for hack-up if the filter preprar finds an 
accaus te description. Heuristics have been used in all 
previous vision preprems to approximate reality Try the rost 
likely interpretation. This may simplify sore problems, but 
sophisticated proprams are needed to patch up the cases where 
the approximation is v/rorp; in my proprsm I have used as 
complete a description as. I could devise with the result that 
the programs are particularly simple, transparent and 
powerful. 


Fifth, because cf this simplicity, I have been able to 
write s. program which operates very rapidly. As a practical 
matter this is very useful for debuppinp the system, and 
allows modifications tc he made with relative ease. 
Moreover, because of its speed, I have teen able to test the 




program cn many separate lire drawings and have thus leer, 
alle to g;ain a clearer understand:nr cf the capabilities and 
ultimate limitations of the program. In turn, this 
understanding has led and should cortinue to lead to useful 


modifications and a greater understanding of the nature and 
complexity of procedures recessary to handle various types of 
scene features. 


Sixth, as explained in the next section, the descriptive 
language provides a theoretical foundation of considerable 
va.lue in explaining previous pork. 


1.7 hiSTOPICAT PFFSPECTIVT 


(re cf the great values of the extensive descriptive 
apparatus I have developed is its ability to explain the 
nature anc shortcomings cf past work. I will discuss in 
Chapter S' how my system helps ir understanding the work of 
Guzman (Guzman I960), Rattner (F.attrer 197C), Huffman 
(Huffman 1971), Clowes (Clowes 1971), and Crban (Crbar 1970); 
and tc explain portions of the work of V/irstor (Winston 1970) 
ard Finin (Finin 1971a, 1971b). For example, I show how 

various concepts such as support can be formalized ir my 
descriptive language. From this historical comparison 
ererges a strikirg demonstration of the ability c f rood 
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descriptions to loth trorc 1 en the range of spylicabili ty cf p 
program, and simplify the propran structure. 


1.8 li PLICATIChS I-CR FUKAI PET CEP8ICN 


lv belief that the rules which gcverr the int emr e tat ion 
cf a line drawing* should be simple is based or the subjective 
ihrressior that little abstraction or processing cf ary type 
seems to be required for me to be able to recognize the 
shadevs, object edges, etc. in such a drawing, ir cases 
vi ere the drawing is reasonably simple and complete. I do 
net b elieve that human perceptual processes necessarily 
resemble the processes in my program, but, there are various 
aspects of my solution which appeal tc my intuition about the 
nature of that portion of the problem which is independent of 


the type cf perceiver. I think it is significant that my 
program is as simple as it is, and that the information 
stored in it is sc independent of particular objects. Back¬ 
up is not necessary in general; the system works for picture 
fragments as well as for entire scenes; the processing time 
required is proportional to the number of lire segments and 
net an exponential function cf the number; all these facts 
lead re tc believe that my research has beer in the right 
directions. 
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2.C CLICK SYIICPSIS 


2his charter provides r crick loch at a 


•ere of the 


tccr .i _ ca.l acre etc cf my work. It provider a. synoosis of wort 
covered mere fully in ry thesis (/.I. TE-271). 


2.1 TIT PI CELT I; 


r what fellows I frequently make a distinction "between 


i:.o scene itself (obiects, table, and shadows) and the 
rftirrj representa tier of the scere as a two-dimerricral line 
diai.jr r . I wilj use the terrs vertex, cdye and surface to 
refer tc the scene features which map into junction, line and 
verier respectively ir the lire drawing. 


of 


Cur first subprcbler is tc develop a language that 
Tiers us to relate these two worlds. I have dene this by 


assifTinr names called labels to lines ir the lire drawing, 
after the manner cf Huffman (Huffman 1971) and Clowes (Clowes 
1S71). Thus, fer example, in figure 2.1 lire sepmert J1—J2 
is labeled as a shadow edge, line 22-J; is labeled as a 
concave edge, lire Ji—JV is labeled as a convex eoVe, line 
cT-uC is labeled as an obscuring edge ard line J12-J13 is 
laoelod as a crack edge. Thus, these terms are attached to 
parts cf the drawing, but they designate the kinds of thirds 
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found ir. the threo-cir ersicnsl score. 


5 hen 


we loch at a line drrwiry cf tl ir sort, we usur 11 v 


err 

cl sil'- 

p 

understand what the 

1 i r e 

dra wiry rerro s ents. 

In 

t. c r r 

s cf a 

label! np sc here e; 

ther 

(1) v/c arc able tc a 

.ssign 

la be 

b..r unic 

uely tc each lire, 

or (2) 

vo car say that ro 

such 

sc or 

e ocul 

d exist, or (; ) v 

e can 

say that although 

it is 


imcssitlr to decide unanb iyucusly whet the Intel cf rn fdye 
rh cult tc, it met te 1? be led with ere eerier of core 
e r reified subset cf the total nurter of labels.. that 


’novloere 


is 


reeded to ere 
If teliro essif rrerts? 


tie the program. to reproduce such 


* 


ruffian end Clowes provided a partial answer ir their 
papers. They pointed cut that each type of function can 
only r. e la tele a ir s, f ew ways, and that if we can saw with 
certainty what the label of one particular line is, we can 
greatly constrain all other lines which intersect that line 
scorert at its ends. As a specific example, if one branch of 
an L junction is labeled as a shadow? edye, then tie other 
branch rust be labeled as a shadow edge as well. 




I crecver, shadows are directional, i.e. ir order to 
ecify a. shadow edge, it must rot only be labeled "shadow" 
but rust also te ranked tc indicate which side of the edne is 


or 


f 
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shadowed. and which side is illuminated. Therefore, rot only 
the type of edge hut the nature of the regions cr each ride 
cen he constrained. 


These facts can be illustrated ir a. jigsaw puzzle 
aralopy, showr in fifure 2.2. Given the five different edge 
types I have discussed so far, there are sever different vays 
tc label any line segment. This implies that if all line 


labels could 1 e arsirred independently there would be 7 


49 


c 


i fferent v/ars to label ar L, 7 = 347 ways to label a three- 

tJ * 


line junction, etc. In fact there are only 9 ways in which 
real sc ne features can map irto Is or a retiral projection. 
Table 2.1 summarizes the ways in which junctions car be 
assifred labelinfs from this set. Ir fifure 2.3, I show all 
the possible labelirfs for each junction tyre, limiting 
rvself to vertices which are formed by ro more than three 
planes (trihedral vertices) ard tc jurctiens of five cr fewer 
lines. Ir Chapter 3 I explair how to obtain the junctions in 
fifure 2.;; I do not expect that it should be obvious to you 
hew one could obtain these junctions. In feneral, for 
clarity, I have tried to use the word labeling to refer to 
the simultaneous assignment of a. number of line labels. 
Labels thus refer tc lire interpretations, and labelings 
refer to junction cr scene interpretations. 
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luKCTIOK 

TYPES 

Number of 

IOSSIBH.1T1ES, 

labeuks 

INOEFEirnBKTtS 

L 

19 

KKESrn 

M3 

FORK 

M3 

T 

313 

PEAK 

2.101 

X 

2101 

K 

2401 

lAixuri 

2101 

K-A 

16807 

K-X 

I6807 


AdTUAL KttWBER 
OTTNXVC& 

JUK-dTIOfcCS CAR ^ 

BE LABELED pEECEKTAGE 


1 

18.1 

3 

2.6 

17 

F.O 

26 

7.6 
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o.z 

37 

l.S 

1Z 

O.S 

21 

1.0 
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QOS' 

1Z 

O.07 
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2.? SCIVIiC TEE 1J EEL ASSICEKJ ET T P.CETFK 


T ^ 

- c 


"be! s 


can te assipred to each line cornert br- a tree 


rearer procedure. Tr terra of the iirrav; ru-rzle aralcsv. 
ii-afire that we have the fcllcvinf iters: 


t * A tcarb v/ith chanrels cut to rerrerert the Dine 
diav-irf; the heard rpa.ee car accept only L nieces at each 
rjrce where the line drawing has an L,‘ onlv /RRCl nieces 
where the lire drevirf has an /RRCV, etc. " Next to" each 
Junction are three bins, marked "Jureticn number”, "untried* 
labels”, and "tried labels”. 


r 

C 


n A full set cf nieces for every space, on the board, 
if tec lire drawirp represented by the board has five Ls then 

there are five full sets cf L pieces with nine pieces in each 
set. 

A set cf Junction number taps marled J1, J2, j* 
bn, where n is tie number cf Junctions or the board. 

A counter which can be ret to ary number betv/eer 1 


/ 


\r c n 


The tree search procedure can then be visualized as 
fcllcws: 


lame each Junction by placinp a Junction number tap 
ir each bin marked ”Junction rumber". 

Step 2: Place a full set of the arrrorriete tyre of 
ir the "urtried labels” bin of each Junction. 

Step Js Set the counter to 1. Prom here or in Kc will be 


used to refer to the current value of the counter, 
the counter is set to 6, then J(No) = 6. 


Thus if 


step 4: Iry to place the top piece from the "untried labels” 
b:r of Junction J(Nc) in beard space J(Nc). There are 
several possible outcomes: 




r o 
£ • C- 


(, i. nr'Tf ~ 

> . V.. ... ' . 


lib 


<A« If the piece er r be pieced (ip. ti e piece re tehee 
all adjacent pieces already placed, if any), then 

A1.. If Nc < n, increase the counter by one and 
repeat Step 4. 


Kc 


/ 2. If 
represent one possible 
this is true then 


n, then the pieces row cn the beard 
label in/" for the line drawing. If 


3 . . 


Write dowr or otherwise rereirber the 


labeling, and 


ii. Transfer the piece in space n bade into the 
n-th "untried labels" bin, and 

iii. Go to Step p. 

i P. If the piece carnot be placed, put it in the "tried 
labels" bin ard repeat Step 4. 

<C. If there are no more pieces in the "untried labels" 
bin, then 


1 ; 


C2. If Nc = 1, we have found all (if ary) possible 
abelings, and the procedure is DONE. 


C2. Otherwise, go to Step 5. 

Step p: Eo ail the following steps: 

i. Transfer all the pieces from the Nc—th 
"tried labels" bir into the Nc—th "untried labels" bir, ard 

ii. Transfer the riece in srace Nc—1 into its 

J.. 

"tried labels" bir, and 

iii* Set the counter to Nc—1, ard go to Step 4. 

To see hew this procedure works in practice, see figure 
2.4. For this example assure that the rieces are piled so 


that the order ir which they are tried is the same as the 
order in which the pieces are listed in figure 2.1. The 
example is carried out only as far as the first labeling 
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obtained by the procedure. There is, of course, at least ore 
ether labeling, namely the one we could assigr by inspection. 
The "false" labeling fourd first could be eliminated in this 
case tv a rr or ran if it krew that HR is brighter thar R1 or 
that R2 is brighter than HI. It could ther use heuristics 
which only allow it tc.fit a shadow edge in ore criertation, 
riven the relative illumination on both sides of a line, 
however, if the object happened tc have a darker surface than 
the table, this heuristic would net help. 


Clearly this procedure leaves many unsolved problems. 

ty 4- V 

Ir general there will be a number of possible labelirys from 
which a rrorrrm must still choose one. V/hat rules car it use 


tc make the choice? Fven after choosiny a labeliry, in order 
tc answer questions (about the number of objects in the 
scene, about which edges are shadows, about whether or not 
any objects support other objects, etc.) a proyram must use 
rules of some sort to deduce the answers from the information 


it hae 


I will argue that what is needed to find a single 
reasonable interpretation of a line draving is not a rore 
clever set of rules or theorems tc relate various features of 
the line drawing, but merely a better description of the 
score features. In fact, it turns out that we car use a 
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psrsirf procedure which involves less ccrr.rtrtz.cr. then the 
tree search procedure. 


o I 7 Tvsy rpcpT* 1- *' r~rv T — OCPT Thm t r' T 
r. •. r ,i a r Jhi c:CrU Jri J.C-1 


he frr J have classified ecyes or3y or the basis of 
H cr ctry (corcave, convex, cbscuriry or rlanar) rrd have 
si hrivid.ee the rlanar class into orach pro shadow sub- 


r> , c* 

C. v ‘ k \ i 


furpese that I further break down each class 


rcccrdiry to whether cr ret each ed.ye err be the hounding 


cf rr object 


Objects err he bounded by otscuriny 


acres, ccrcavc edyes, ard crack edyes. Fiyure 2.5 shows the 
results cf apperciny a label analcyous to the "obscuring 


ec>e" rark tc crack and concave edres. This rrrrcach 


n is 


'Zi. ilrr tc ere first proposed by Freuder (Freuder 1971a). 


lach resion can also be labeled as belcnyiry tc one of 
ti e three followiry classes: 


Illuir.inated directly bv the liyht source 


£P — A projected shadow reyicn; such a reyicn would be 
illurirated if re object vere between it anc the liaht 

w w. 

source - 
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C 


S A S6. If— rhc?.C CV6d T&fZ CF ; SUCh £ region 1.S crier ted 


away from the light source 


fiver these classes, I car defire new edge labels which 
O.J..SC include information about the lighting or both sides of 
tie edge* hetice that ir this way I can include at the edge 
level, a very local ievel, irforraticn which constrains all 
ecres bounding the same two regions. Put another way. 

%J 7 

wherever a lire err be assigned e. single label which include 


s 


th i s 


lighting irforraticn, ther a program has rowerful 


ccrstraints for 


the junctions which can appear around either 


cf the repions which bound this line. 


I ipure m.6 is rra.ee up of tables which relate the region 
illumination types which can occur or both sides cf each edge 
type. For example, if either side of a concave or crack edge 
is illuminated, both sides of the edge must he illunirated. 


These tables car be used to expand the set cf allowable 
junction labels; the new set of labels can have a. number of 
entries which have the sair.e edge geometries but which have 
cafferent region illumination values. Tt is very easy to 

tJ 

write a program to expand the set cf labelings; the 


principles oj. its operation are (1) each- region in a given 
junction labeling can have only ore illumination value of the 
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three, and (2) the values or. cither side cf each lire of the 
juretion must satisfy the restrietiers ir the tables of 

<-> * r\ r 

iicure a.(. 


/n interesting result of this further subdivision of the 
.lire labels is that, with four except lore, each shad ov¬ 
er u sir r jur.ction has only ere possible illumination parsing, 
as shewn ir fiyure 2.7. Thus whenever a scene has shadows 
are vh never a urofTSir. car find a shadow causiry junction in 
such a scene, it car erectly constrain all the lires and 
reyiors which rale up this junction. Tn fiyure 2.7 I have 
also : arked each shadow edye which is part of a shad ow¬ 
es us ir a -unction with an "L" if the arrow on the shadow edye 
pcirts counter—clockwise and- an "R" if the arrow points 
clockwise. he "I” shadow edye can match an ”K" shadow ecye, 
correspondiry to the physical fact that it is impossible for 
a. shadow edye to be caused from both cf its ercls. 


r i; ch 

a scene, 

it 


ore which 

F:a 

p ] r c 

: pxkec c 

ach 


There are two extreme possibilities that this 
partitioning ray have on the rumber of juretion labelings now 
needed to describe all real vertices: 


(1) Iach old juncticr label which has n concave edges, m 
crack edyes, p clockwise shadow edges, a counterclockwise 
shadow edges, s obscuring edges and t convex edges will have 
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tc be replaced by (20) (6 f* (3? O 'p' (9) S (sf" net: jurcticns, or 


(2) Each ole junction will five rise to only ere new 
iurcticn (as in the shadow-carsirf junction cases). 


Jf (i) wore true then the partition would be worthless, 
since no rew inforration could be faired. If (2) were true, 
the situation would be greatly improved, since in a sense all 
the much more precise information was implicitly included in 
the or'final junctions but was net explicitly stated, 
because the information is new more explicitly stated, many 
matches between junctions can be precluded; for example, if 
in the old scheme sore line segment LI cf junction label 01 


ccu.ld have been labeled concave, as cculd line segment L2 of 
junction label Q2, a line joining these two junctions cculd 


■eve teen labeled concave. hut in the rev; scheme, if each 
iurcticn label gives rise to a. single new label, both LI and 


L2 would take on one of the twenty possible values for a 
concave edge. Unless both II and L2 gave rise to the same 
new label, the line segment cculd rot be labeled concave 
usins 01 and 02* The truth lies somewhere between the two 
extremes, but the fact that it is not at the extreme of (1) 
means that there is a net improvement. In Table 2.2 I 
compare the situation now to cases (1) and (2) above and also 
tc the situation depicted in Table 2.1. 
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I have else used the better descriptions to express the 
restriction ihat esc} scene is assured tc be on. a horizontal 
table which h? s nc hoJ.es in it a.rd wb.ich is large encuph tc 


-n; l 1 
J. j. J.J- 


the retina. This rears that anv lino segmert which 


separates the background (table) from the rest of the scene 
can only be labeled as shewn in figure 2.£. because of this 
fret the rurber cf luncticn labels which could be used to 
label junctions on the scene/background boundary/ can be 
greatIv restricted. 


The value cf a better description should be immediately 
apparent. Ir the old classification scheme three out of the 
seven lire labels could appear or the scene/background 
boundary, whereas in the new classification, only seven out 
of fifty labels can occur. Korcover, since each junction 
must have two of its line segments bounding any region, the 
fraction cf junctions which can he cn the scene/background 
boundary has improved roughly from (3/7)(3/7) = 9/49 = 17.4 % 


tc (7/57)(7/57) = 49/3149 


1.6/. The results of 


improvements will become obvious in the next section. 


these 
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2. A PI.CGE/ Mill G CGKSECUENCFS 


here are ro mrny rossible labels for each tyre of 
•^u.rcticn that I decided to tea-5n -programming a Dateline 
v" ate; by writ inf a sort cf filtering program to eliminate as 

r. ' tj <w . O v X ^ 


r 1 ' iurctior labels as -possible before beginnins a tree 

' *J X 1 - -1- 


J . i. £ 


E(.:fTCC prcc 


lure 


r • h e 


filter rrocedure derends on the fcllcvinf' 

-i- A, 

observation, fiver in terns of the jigsaw puzzle analogy: 


fuppcse that we have two junctions, J1 and J? which are 
joined by a. line segment L-J1-J2. J1 and J2 are 
represented by adjacent spaces on the board and the 
rcssible labels for each junction by twc stacks of 

Kov for any piece h ir Jl's stack either (1) 
is a latching piece H in J2's stack or (2) there 
piece- If there is re matching piece for M 
then V: can be thrown away and reed never be considered 

rcssible -unction label. 
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Ihe filter procedure below is a method for 
svstei a.ticall-'- eliminating all junction labels for which 
there can never be a match. /II the equipment is the same as 


that used in the tree search example, except that this time I 
have added a card marked "juretier modified" on one side and 
"ro junction modified" on the other. 
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utep 1: Put a junction numTer tag letween 1 rrc n in 
each "juretier number" bin. Place a full act cf nieces 
in the "untried labels" lin cf each -'unction. 


r' 


Set the ccu.rtcr to i c 
co tl at it reads "no iuncticr 


Step p: Check tie value cf Ic: 


A 


: 1, and rla.ee tl e card 
U ice.". 


If he = n + 1, and the card reads "rc "unction 
uodified" then pc to SUCCEED. 


r 

i 


If lie = n +1, and the card reads "-'unction 
icciiied 1 tier- [c tc Step 2. (At least ore piece was 
'thrown avay on the last pass, and therefore it is 
cssible that other pieces which were kept only because 

this piece v/as present will now have to be thrown awa^ 
also.) 

C. Otherwise, go tc Step 4. 

itep 4: Check the "untried labels" bin of -'unction 

S (lie;: ' 


If 


A. If there are no pieces left in 
untried labels" bin, then 


the ife-th 


A1. If there are ro pieces ir ti e Nc-th 
"tried labels" bin, go tc FAILURE. 

A2. Otherwise, transfer the pieces from the 
tc—ti "tried labels" bin back into the lie—th "untried 




labels" bin, add 1 tc the counter (lie) ard gc tc Step 

E. If there are pieces left ir the Nc-th "untried 
labels" bin, take the tep piece frem the bin ard rEcce 
it in the board, and go to Step D. 

Step 5: Check the spaces adjacent tc space lie: 

A. If the piece in the Nc—th space has matching 
pieces in each neighboring- junction sra.ee, transfer the 
niece from space Tie into the Nc-th "tried labels" lin, 
and transfer the pieces fror the neighboring srsces and 
the neighboring "tried labels" bins back intc their 
"untried labels" bins. 

F. If there, are empty neighboring spaces, then 


r T'pr- '( ' 
i — ' j v- x 
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El. If there are no more function. 0 in the 
neighboring "untried, labels" bine which could fit with 
the riece in space lie, ti er that riece ir not a possible 
.label. Thrcv it away, and arrange the card to read 
"junction modified" if it doesn't already. 

22 . Try pieces from the neighboring "untried 
labels" piles urtil cither a piece fits or the pile, is 
exhausted, ard then pc tc Step 5 again. 


aiCCrFE: The pieces 
each juretier have 
constitute the outnut 


in the "untried labels" bins of 
passed the filtering routine and 
of this procedure. 


FAILURE: There is nc wav to label 
current set cf pieces. 


he scene riven the 


Ir the program I v/rcte, I used a somewhat mere complex 
variation of this procedure which only requires ore pass 
throufh ti e junctions. This procedure is similar to the one 
used tc venerate figure 2.$, and is described belev. 


I hen I ran the filter program on sore simple line 

JL. ** 

drawirgs, I found to my amazement that the filter procedure 
vieldcd urioue labels for each junction in most cases! In 
fact in every case I have tried, the results of this 


filtering program are the same results which would b 


0 


obtained by running a. tree search procedure, saving a.ll the 
labelings produced* ard combining all the resulting 
possibilities for each jurcticn. In other words, the filter 
program in general elimirates all labels except those which 


o *v 
c a. 


e mart cf seme tree search labeling for the entire scene. 

-J. v..v 
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It ir not obvious that this should be the case. 


J- or 


c'T.ifT-i o, if this filter rrooedure ir err lied tc the sirrle 
line drawing shown in figure t .A using the clc set of lot els 
fiver in figure it rredrees the results shorn ir firure 


' o 


In this figure, each junction has Ir heir attached which 


would net he t art of any total labeling rrocuced h' 1 '- a. tree 

j *j ' j > 


c r r c. 


ivreti ens 


This f i ru.re is ot tai 
in numerical order and: 


tv roirr- thrci’rh the 


(1) attaching tc a juretier a 13 labels which do not 


conflict with junctions rrevic-us3 ' r assigned: i.e. 


if it is 


hr own that a 1ranch rust he 3ate]ed from the set S, do not 
attach- ary junction labels which vculc require that the 
branch be labeled with an element not in S. 


(2) lookina at tie neighbors of this junction which have 
already beer labeled; if any label does rot have a 
corresponding assignment for the same branch, then eliminate 
it. 


(3) V/herever any label is deleted from, a junction, look 
at all its neighbors in turn, and see if any cf their labels 
can be eliminated. If they can, continue this process 
iterativelv urtil no more charges can be made. Then go or to 
the next iu.notion (numerically). The junction which was 

kJ ' t / v/ 
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ccirr Is.!elec (as ir step (i)) Pt the tire 
eliminated (struct cut in the firure' 


lr tel wps 


eiir iratec latel in finure 2. c . 


is noted next tc each 


ee fact that these results can he rredveed tv the 


filterinn profram ssvs 


c 


ftcb. t d eai 8, c cu t 1 ir 6 d rp,wi n/^s 


sorer,' ted by real scenes and else a.hcut the value of rrecise 
descriptions. There is sufficient local irforraticn ir a 
lire draviny so that a program can use a procedure which 
recuires far less computetion than dees a tree search 


rrocce ure. 


- c ree why this is sc, notice that if the 
description the program uses is need enough, then many 

functions rust always be given the same unique latel in each 

txee search solution; the filtering program needs to find 

oicn latel only once, while a tree search procedure rust go 

thrciy h the process of findinf the sare solution or each pass 
through the tree. 


II 


Cuite remarkably, all these results are obtained usins 
only the topology of line drawings plus knowledge about which 
region is the table and about the relative brightness of each 
region. No use is made (yet) of the direction of line 
segmerts (except that sore directional information is used to 
classify the junctions as AFRCV/s, FCRKs, etc.), nor is any 
use race of the length of line segments, microstructure of 
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edges, lifhtirg direction or ether potent:ell"- useful cue? 


2.5 KINDLING I AD 




i -Li i 


No for I have treated this subiect as though the prorram 
vculd always be .river perfect data. In fact there are rany 
tyres cf errors and degeneracies which occur freruert.lv. 
Sene cf these can be corrected through use of better Dine 
finding programs and sene can be elirrinated by us inf stereo 
irfen. aticn, but I wculd life tc shew that the prof ram can 
handle- various prebiers by sir pie extensions of the D ist of 
junction labels- Ir no case do I expect the program tc be 


itle to sort cut scenes that people cannot easilv understand. 


jwc cf the rest common types of bad data are (1) edpes 
missed ertircly due to equal repion brightness or both sides 
of the edpe, and (2) accidental alignment cf vertices and 
lines. figure 2.10 shows a scene containinp instances of 
each type of jrobler. 


The program handles these problem junction 


s 


ov 


generating labels for them, just as it dees for normal 
junctions. It is important tc be able to do this, since it 
is in general very difficult tc identify the particular 
junction which causes the prepram to fail tc fird a parsing 
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C h r v 1: e r 7 t r ea t s 

a.ccidental a] if nr ent. 

crr. i. or tvies which are 

*/ 

consider three birds cf 


r nuirher of different 
Figure 2.11 shows three 
included in the pregrar's 
accidental alignrent: 


tyyes of 
cf the rost 
repertoire; 



cases where 


& 


because ar 


edfe obscured 


vertex arparentlv has 
ty the vertex arrears 

^ J. ^ 


an extra line 
to he part of 


the vertex (see figure 2.11a), 


(2) cases where an edge which is between the eye ard a 
vertex arrears to intersect the vertex (see figure 2.11h), 


arc 


(3) cases where a shadow is projected so that it 
actually does intersect a vertex (see figure 2.11c). 
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I have rot attempted 1c systematically incit'de all 
l. isv.ii y line j ossj.tilj.ties, out have only included lalels for 
the j ost conn on typer of rissiny lines. J recuire that anv 

‘V 

rissiny line Te ir the interior of the scene; no line on the 

scene/bacl yround boundary can he r issiny. I also assure that 

ell objects hove approximately the sere reflectivity on aU 

surfaces. Therefore, if a convex line is missirr, 7 assure 

t. < u citncr both sides of the edye were illurinated cr that 

tctr were shadowed. I have rot really treated missiry lines 

ir <;•. complete enouyh way to say much stout them. There will 

have to he facilities ir the program for filliry in hidden 

surfaces and hack faces of objects before miesinr lines can 

be treated satisfactorily. 


In general the program will report that it is urahle to 
label a scene if more than a few lines are rissiny and the 
missiry line labels are not included in the set of possible 
junction labels* This is really a siyn cf the power of the 
proyrar, since if the appropriate labels for the rissiny line 
junctions were included, the prorram would find them 


uniquely 


As an example, the simple scene in firure f.12 


cannot be labeled at all unless the rissiny line junctions 


are ireluded. 
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Ricici-: crjfnt/ticit: 


Joy ions cr.r he esrirrsc lc heir vhich rive ciart2 zed 


cirrmcrr. 


■ i r- f 

. j. 


vrluer for rer icr. orier.tr t lore ir three 
jjsc^.r err hr adr'ed tc the ;urcticr Irbrlr in very ru.ch the 
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T - cy that the rerier. illur irr.ticn vrlrer vere added 
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Clearly there are considerable obstacles to be dylfcoine 
ir ext end inf this verb to general sceres. For si Erie curved 


o;jeers 


fucn a 


cylinders, spheres, cones, and ccnic 


sections, there should be no particular problem ir using the 
tyre of program I have writ'ten. (For a suite different 
approach to the handling of curved objects, see Hern 1970.) I 
also believe that it vill be possible to handle screw!at more 
general scenes (for instance scenes containing furniture, 
tools and household articles) by approximating the objects in 
tier by simrlzfiec "envelopes” which preserve the stops form 
cf the objects yet which can be described in terms life those 
I have used. In r.y estimation such processing carrot be done 
successfully until the problem cf reconstructing the 
invisible portions of the scene is solved. This problem is 
intimately corrected with the problem of using the stored 


description of an object to guide the search for instances of 
this object, or similar objects in a scere. The ability tq 

1 ■ 1 ^ 

-label a line dfWiftg' w 'in the manner I desorlb#: greatly 


simplifies the specification and hopefully v/ill simplify the 


r, VV e ■ • 
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A VISION POTPOURRI 


Tin Finin 



June 1972 


AESTRACT 


Thie paper discusses sore recent changes and additions to the 
vision system. Among the additions are the ability to use visual 
ieedback when trying to accurately position an object ard the 
a liity to use the arm as a sensory device. Also discussed are 
some ideas and a description of preliminary work on a particular 
sort Oj. higher level three—dimensional reasoning. 


This paper was originally published as Vision Flash 26. 
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JIGGLING A BLOCK INTO PLACE 

The vision system can now use visual feedback when trying to 
accurately position a block. This is done without a costly 
rescanning of a significant portion of the scene by using our 
knowledge of where the block should be to direct the eye. The 
basic idea is to determine the block's actual location by looking 
for certain key vertices using a circular-scanning vertex finder 
developed by Winston and Lerman < Vision Flash 24 >. 

When placing a block the arm sometimes makes positional 
errors up to half an inch and rotational errors of about 10 
degrees. These errors are caused by poor hand placement due to 
hysteresis and general slop in the arm's joints and by poor 
information about the brick's initial position and dimensions due 
to a distorted line drawing. Although these errors can be 
disastrous in delicate tasks such as stack-building, they are 
small enough to allow us to use the scheme described below. 

The organization of the theorems is shown in figure 1. TC- 
JIGGLE, the top level theorem, first calls TC-FIND-BODY whose 
goal is finding the actual location of the just moved brick. 
This is done by locating a three-vertex 'skeleton' on either the 
top or bottom of the brick, examples of which are shown in figure 
2. Candidate skeletons are suggested by the theorems TC-LOCKFOR— 
TOP, TC-L00KF0R-B0TT0M, and TC-LOOKFOR-SKELETON which predict the 
locations of vertices and decide whether they should be visable. 
TC-FINDBODY then locates the three vertices comprising the 
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skeleton with the circular-scan vertex finder and calculates the 
true position of the brick. If it fails to find one of the 
vertices, it asks for another skeleton and tries again. 

Once the location of the brick is found, TC-SHIFT-BODY 
calculates the positional and rotational errors and, if they are 
greater than a tolerance, corrects them through a call to TC- 
MOVE-GENTLY. This theorem differs from the usual TC-MOVE in 
calling the arm with GRASP and UNGRASP commands instead of PICKUP 
and DROP. PICKUP and DROP raise the arm several feet above the 
table when moving to avoid obstacles, whereas GRASP and UNGRASP 
lift the hand less than an inch (using the wrist) and thus, 
hopefully, are less prone to error. 

The most difficult part of this jiggling procedure is 
determining which vertices of a brick will be visable and not 
obscured by other objects. V/e must also avoid looking for 
vertices which are adjacent to others already in the scene, for 
example the vertices where two bricks are aligned. Such 
situations may confuse the vertex finder and cause it to find the 
wrong vertex. Since these theorems are written to work in the 
context of a copying task, they use information about the model 
scene that is being copied* For instance, before TC-LCCKFCR-TCP 
looks for any vertex on the top of a brick it must either find 
that: 

1. The top of the matching brick in the model was 

completely visable, or 
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TC-JIGGLE 




TC-SHIFTBODY 



TC-LOOKFOR-TOP TC-LOOKFOR-BOTTOM TC-LOOKFOR-SKELETON TC-MOVE-GENTLY 

GRASF UNGRASP 



l 


TC-CLEARVIEW 


TC-CHECKPOINTS 


l 



TC-NEIGHBORING-BODIES 


TC-GOOD-CHECKPOINT 


TC-SKELETON 


FIGURE 1 



FIGURE 2 
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2. All bricks which could be adjacent to the one in 
question are either below it or have not yet been placed. 
The theorems lean toward conservatism in accepting vertices as 
good candidates to look for and will reject all of them in some 
cases. 

One exciting possibility for further work is the 
incorporation of a model of the hand. With it we could adapt the 
system to avoid vertices occluded by it, doing away with the 
necessity to release the brick and withdraw the hand. This would 
result in a more dynamic and accurate feedback system. 
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OUR ROBOT HAS A HMD, TOO 

Until now the vision system has made no use of its arm in 
getting information about the world. We now have a limited 
ability to reach out and touch in order to disambiguate some 
scenes, using a new arm primitive written by Jerry Lerman. 

Sending (TOUCH X Y) to the arm causes it to position itself 
above the point (X,Y), slowly descend until it touches something, 
and report its final height. An optional third argument can 
specify a maximum height at which something is expected, allowing 
the arm to rapidly drop to this height and then more slowly feel 
its way downward. 

A series of theorems have been written which actively use 
the arm as a sensor and other theorems have been taught to use 
them, resulting in the system network shown in figure 3. With 
these theorems we can now handle scenes such as the pedestal in 
figure 4. In this scene we can't determine the tallness of B1, 
since it could touch the bottom of B2 near the front, the back, 
or somewhere in between. As a result, we can't get the 
dimensions and location of B2 either. 

We can however determine the location of B1 in the X-Y plane 
(through TC-Fim)-LOCATICN-BClTOK). Moving the arm down over this 
spot until it touches the top of B2 gives the altitude of B2's 
top. With this information we can calculate the location and 
dimensions of both bricks. 

Previously, when we wanted to—find the location or 
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dimensions of a trick we had to find its altitude above the 

table. If it was not resting on the table, we had to find the 

dimensions of its supports, necessitating knowing their altitudes 

above the table, etc...... V/e recurse downward until we reach the 

table or fail by hitting a brick for which no tallness or 

altitude can be found. With these new theorems we have another 

alternative: recursing upward until we find a brick we can touch 
with the arm. 

One problem is that we aren^t working with a very good three 
dimensional model. TC—TOUCHTCP is the theorem which tries to 
touch the top of a brick. Checking first that there is nothing 
above the brick, it tries to touch it above the center of one of 
its supports. The brick could, however, not be above this spot 
(as in figure 4b) causing the arm to miss it. One precausicn 
that TC-TOUCHTOF takes is calculating the minimum height to 

expect the top of the brick. If it touches something below this 
height, it assumes it missed. 
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FIGURE 3 
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TC-FIND-SUFFORTS 

TC-FIND-SUFPORTS and its related theorems have been modified 
to handle situations with which they previously could not cope. 
Figure 5 show3 the new organization of this part of the system. 
The strategy of TC-FIND-SUPPORTS was to take each object below 
the brick in question (found through TC-FIND-ABOVE-1 & -2) as a 
support candidate. The altitude and tallness of each candidate 
were found and summed. The object or objects (if there were 
several with nearly equal combined altitude and tallness) with 
the largest sum were then taken as the actual supports. This sum 
was then asserted as the altitude of the supported brick. The 
theorem failed if it could not find an altitude or a tallness for 
one of the bricks below. 

The new TC-FIND-SUPPORTS works in much the same way, but has 
been modified to handle many cases where the tallness or altitude 
of an support candidate can not be found. In such cases it 
determines the minimum height that the top of the candidate could 
have. 

It will also yield useful information in cases where it is 
still ambiguous which objects support another. Before failing it 
makes assertions of the form: 

(B1 may-be—supported-ty B4) 

(El may-be-supported-ty B7) 

(B1 has-minimum-altitude 4.12) 

These assertions can later be used by other theorems with more 
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real world knowledge to clarify the scene. For example, we might 

call on a theorem which knows about stability or one which can 

recognize a table top and legs to decide who is doing the 
supporting. 

Two auxilliary theorems are used, TC-ArD-TC^-SUFPORTS-1 and - 

2, which contain some 3-D knowledge. TC-AED-TC-SUPPORTS-1 looks 

for a marrys relation between the brick in question and a support 

candidate. If one is found, the theorem reports that it must be 

a support (assuming gravity and no glue). TC-ADD-T0-SUFP0RTS-2 
is explained below. 

The capabilities of the new TC-FIND-SUPPOPTS are best shown 

in the scenes in figure 5 • For each of these scenes the old TC- 

FIND—SUPPORTS would simply fail, leaving no assertions in the 

data base. Figure 5e is particularly interesting , showing the 

application of some three dimensional reasoning. On this figure 

TC-FINIV SUPPORTS first calls TC-FIND-SUPPORT-CANDIDATES which 

reports that B2 and B3 are likely support candidates and that El 

must have an altitude of at least T. TC-ADD-TO-SUPPOPTS-1 then 

finds that E2 marrys B1 along B1's bottom edge, implying that E2 

must support B1 and that B1 has an altitude of T. TC—ADD—TO— 

SUPPORTS—2 is activated and notes that El's altitude is now known 

to be T. Discovering that the minimum tallness of B3 is also T 

(within an epsilon) it asserts that B3 must also marry B1 and be 
a support. 
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TC-FIND-ALTITUDE 


TC-FIND-SUPPORTS 
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TC-FIND-SUPPORT-CANDIDATES 


l 


£ 

TC-TOPHEIGHT 



TC-FINDABOVE 1 
TC-FINDABOVE 2 


TC-FINDABOVE 3 


TC-ADD-TO-SUPPORTS 


TC-ADD-TO-SUPPORTS 


TC-MARRYS 

a*TC-LINE-BELONGS-TO-REGION 


FIGURE 5 




B 3 





Assertions made by 
TC-FIND-SUPPORTS: 

(B1 is-supported-by B2) 
(B1 is-supported-by B4) 
(B1 is-supported-by B5) 
(B1 is-supported-by B3) 
(B1 has-altitude t) 







Page 160 


(B1 is-supported-by B2) 

(B1 has-minimum-altitude t) 


(B1 is-supported-by table) 
(B1 has-minimum-altitude t) 


(B1 may-be-supported-by B2) 
(BX may-be-supported-by B3) 
(B1 may-be-supported-by B4) 
$B1 has-minimum-altitude t) 


(B1 is-supported=by B2) 
(B1 is-supported-by B3) 
(B1 has altitude t) 
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CHANGES TO TC-SKELETON 

TC-SKELETON has teen changed to return a little more 
information if it can't find a complete skeleton for a trick. 
When TC—SKELETON fails to find a line of a particular type it 
tries to find the longest line fragment of that type and makes 
partial-skeleton assertions of the form: 

(B3 type-two * (VI V2)) 

(B3 has-partial-skeleton V4 VI * (VI V2) VI V14) 

Figure 6 shows some examples of partial skeletons. 

This partial skeleton information is used by other theorems 
which hypothesize what the rest of the brick may be like. From 
it we can get a handle on some three-dimensional information such 
as a brick s orientation, its minimum dimensions, and some idea 
of its location^ Since other theorems make hypotheses that 
complete parts of the skeleton, an antecedent theorem has been 
added to keep the skeleton assertions up to date. 

FIGURE 6 


(B4 has-partial-skeleton * (VI V2)*(vi V3) V2 V4 
(B1 has-partial-skeleton VI V2 VI V3 * (V3 V4)) 




'TC-LINE-BELCNGS-TO-REGICN 

A new theorem exists which will determine whether a line is 
.physically associated with a particular region in a picture. 
This question crops up in numerous places in the vision system: 
finding the skeleton of a brick, finding a brick's tallness, 
finding bottom lines, deciding whether a face of a brick is 
vertical or horizontal, etc. In each of these places several 
heuristics were employed to find lines which 'belonged' to a 
region. TC—LINE—BELONGS-TO—REGION is a collection of these 
heuristic and several- new ones which can be called whenever 
needed. It makes assertions of the formt 

(L-V1-V2 belongs-to R4) 

One of the heuristics is that an interior line of a body 
belongs to both regions it bounds, so that TC-LINE-BFLONGS-TO- 
REGION should not be used on self-occluding bodies. In general, 
the heuristics are applied conservatively, sometimes calling on 
the theorem TC—OCCLUSION—1 to look for supportive or 
contradictory evidence. Consequently some lines will not be 
recognized. Having this information at hand makes many tasks 
much easier. For example: 

* A region is a vertical face if there is a vertical 
line which belongs to it. 

* Two objects marry if we find a line which belongs to 
a region from each body. 
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How many things can you find wrong with this picture? 

t 

In a picture such as the one above where one brick obscures 
the bottom of another, we can extract some information on the 
whereabouts and size of the obscured brick. Examine the scene in 
figure 7. B2 could be a very tall brick which was touching B1, 
or a shorter brick far behind B1, or anywhere in between. It's 
ambiguous. Knowing the range of possible heights and resulting 
locations of the obscured brick will be quite useful to other 
thoerems which try to decide what the situation really is. To 

get quantitative three dimensional information we use the 
procedures described below. 

To get the maximum tallness ofB2 we assume that it is 
directly behind and touching B1. Assume for the moment that the 
ends of B2's three vertical edges (b, c & d) touch the edge a-e. 





These three points would then have an altitude of h (the 3-D 
tallness of B1) from the table. From this we can get their 3-D 
coordinates and the resulting lengths of the three verticals. We 
then take the shortest of these lengths as the maximum tallness 
of E2. This corresponds to selecting the vertical ending in c as 
the only one which could touch El. 

To get the minimum tallness we assume that B2 is far behind 
B1 and its bottom edges just barely obscured. For the moment 
assume all three points are on the table. We get their 3-D 
tallness coordinates and calculate the lengths of the verticals. 
The maximum of these three lengths gives us a minimum tallness 
for B2. Taking the longest corresponds to selecting the point b 
as the only one which could actually be on the table. 

The problem with the picture on the previous page is that, 

» 

assuming the ocscured object is a brick, its apparent minimum 
tallness is greater than its apparent maximum tallness. 

A further refinement of this heuristic is shown in figure 

8. In this picture, to get the minimum tallness of the obscured 

♦ 

brick we assume that the points a and b rest not on the table, 
but on the regions R1 and R2 respectively. We must check of 
course, that these regions are not vertical, as in figure 8b. 

The same situation exists for the inverted case, the 
pedestal in figure 9- If we assume that B1 supports B2, we can 
put upper and lower bounds on the height of B1 (and the resulting 
altitude of B2). We know that the top of B1 must touch the 
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bottom of B2 somewhere near the front, the back, or in between. 
Getting the minimum tallness is trivial, just measure the length 
of the three vertical edges of El and take the largest of these. 
This corresponds to a situation where B1 and B2 marry along their 
front edges. To get the maximum tallness we start off by 
locating the visible bottom edges of B2 and predicting where the 
others should be. We then extend each vertical until it 
intersects one of the back bottom edges of E2. After calculating 
the 3-D lengths of these verticals we take the shortest as being 
the upper bound on the height of B1. 

Considering the stability of such structures would of 
course lead to more refined upper and lower bounds. 
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ONE KIND OF THREE-DIMENSIONAL REASONING 

Everyone agrees that future advances in computer vision will 

come with the incorporation of 3D knowledge and reasoning. One 

type of 3D reasoning we could add is shown in figure 10. In 

figure 10a the average human viewer (trusting and non-cynical) 

sees two identical bricks supporting a third , even though El 

could be longer or shorter than B2. Because B1 and B2 serve the 

same function (supporting B3) , have the same orientation, and as 

far as we can see, could be the same size, we easily assume that 

they are. Similarly , in figure 10b , we see B4 as being the 

same size as the other three standing bricks. Here the evidence 
is: 

* There is a preponderance of tall, thin standing bricks 

* El, B2, and E3 form a row, which would include B4 

if it were the same size 

* B1, B2, E3, and B4 all have the same orientation 

* E4 could be the same size as B1, B2 and B3 

Humans do this hypthesizing and filling in of details to a 
great degree. It is an integral part of perception, as it would 
be impossibly costly (in time and effort) to try to disambiguate 
everything we see. Teaching the machine to do such hypothesizing 
would be a natural way to incorporate some three dimensional 
reasoning and to enable it to consider the global context of the 


scene. 
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A similar form of reasoning was pointed out by Winston in 
his thesis < Learning Structural Descriptions from Examples , AI 
TR-231 >. In figure 11, B2 appears to he a wedge, while we see 
£4 as a brick, even though they show the same arrangement of 
lines and faces. In both cases since the two objects form stacks 
and are exactly aligned we first assume that they are identical. 

We might object that this presupposes an orderly scene , one 
which is carefully set up or contrived. However, the world we 
humans create, and which robots may inherit, is just such an 
orderly, contrived world. Even in the mini-world of plain white¬ 
faced blocks we use in our present research we tend to build 
little arches, stacks, and other structures containing identical, 
interrelated parts. In the larger world of human construction 
this orderliness is more apparent. We tend to build things which 
are symmetric and unsurprising in details. Complex objects such 
as buildings, electronic circuits, and cars are built using 
smaller identical parts (e.g. standard-sized windows, resistors, 
bolts). Who would suppose that the wheels on one side of a car 
are any different than those on the other? Long hallways usually 
have identically dimensioned doors uniformily spaced. The legs 
of a table or desk are nearly always the same. 

We might also object that a robot would do better to spend 
his time trying to disambiguate a scene by removing some 
obscuring obstacles, by walking to one side, or by reaching out 
his hand and touching. In many cases however, our robot may find 
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it impossible to change his viewpoint or to interact with the 
scene. Even more importantly, the ability to do this sort of 
reasoning would allow him to have some expectation as to what 
will most probably be seen if obscuring objects are removed, or 
the viewpoint is changed* The robot can then quickly test his 
previously formed hypothesis. If this verification fails, he can 
flush the hypothesis and examine the scene more carefully. 
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The following pages describe initial work in creating a 
system of theorems to do just this sort of reasoning. A skeletal 
system has been implemented which will handle a number of the 
simpler situations (such as those with fairly obvious group type 
relationships — rows, stacks tables, arches, etc.). 

Whenever we can't find the complete dimensions of a badly 
obscured object, as in figures 10a and 10b, we check for evidence 
that this mystery object might be just like some other object in 
the scene. Typical of the evidence we look for are: 

* Chains of relations (rows, stacks, walls) 

* Identical relations to a third object, or functional 

similarities (e.g. supports of an arch, table legs) 

* A preponderance of identical objects in the scene 

* Exact alignment with another object 

* Other relations and properties such as attitude, 

nearness, placement, marrys, abuts, etc. 

/ 

If we find such a candidate we check for contradictory ev¬ 
idence. Things we check are: 

* That the known dimensions of the obscured object 

agree with the corresponding dimensions in its 

supposed double. 

* The unknown dimension(s) lie within the 
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apparent maximum and minimum "bounds 
we have calculated. 

* That the hypothesis yields no colliding objects (i.e. 
those occupying common space). 

* That ve do not contradict any previous assumptions 

we have made (e.g. support). 

In some cases we need not require an object which might be 
an exact double, but only one which implies a similarity that 
might resolve the hidden dimensions. For example, in figure 12 
we hypothesize that that the two stacked bricks have the same 
depth, even though their heights are different. 


FI SORE 10. 

£ 

* • 

zhi 

If we find any contradictory evidence, we reject the 
candidate and can lcok for another. If no contradictory evidence 
is found, then we make our calculations and tentatively assert 
the dimensions and location of the object given our hypothesis. 
This is done with pointers to the theorems which suggested this 
hypothesis so that we can reconstruct our reasoning if needed. 

With this information we can proceed to analyze the scene 






and hope that we have not been tricked. Alternatively ve can try 
to verify the hypothesis by some other means. Tor instance, ve 
could create a daemon theorem to lurk in the background waiting 
for some of the obscuring objects to be removed. When they are, 
the eye can be asked to verify critical lines or vertices 
suggested by our hypothesis. For our paradigm arch case, as soon 
as the top brick is removed, we can look for the top back 
vertices of the mystery brick. If the vertices aren't found near 
their proposed locations then the hypothesis is flushed. In such 
a case we could use the eye to more completely scan the area to 
resolve the problem. In other situations we might be able to use 
the hand as a sensor to verify the hypothesis or find out what 
has gone wrong. 
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Foreword 


Thus material is rot presented as a discovery but rather ss 
a journey or the path laid down by Professors Minsky, Pepert and 
Vanston in Vision. Please ignore any dogmatic tone which may 
appear in the efforts to overcome my natural tendency to the 
opposite "I think, perhaps, maybe..." school of rhetoric. 



Abstract 

A proposed program of research in Machine Vision is 
described. The first section, the Scenario, lays out succinctly 
and infernally the aims of the project and summarizes the 
background of the work, the problems to be faced and the approach 
to be taken. An extended description of the Problem follows with 
its background and its formulation for the present work. We next 
discuss the Artificial Intelligence Issues involved, ard the 
Approach we will take. Benchmarks are laid down for specific 
progress. Our research interests are once again carefully 
formulated in the Summary. 


Scenario 


Picture the following scenario. 

We walk into the street, and hijack the first passerby. We 
have him go home, retrieve his toolbox and return to the A1 
Robot. He grabs a handful of tools and dumps them ir front of 
the eye. The Robot identifies these objects. 

The key element here is flexibility in dealing with real 
objects in real situations (surreal perhaps in this instance). 

The flexibility to deal with reality will be the basic aim 
of our research. 

Why has previous work in visual recognition often been sadly 
lacking in this regard? Why does this effort have a better 
chance of success? 

Consider first some recent contributions. 

1. Minsky and Papert—The heterarchical approach essential 
to the flexibility sought (4,5>6,7). 

2. Winston—Network structure, data driven control, basic 
structural description, learning by failure, grouping, 
quantitative analysis of relational concepts (11,12,13). 

3- Shirai—A.I«, and vision research in particular, as an 

experimental science, as demonstrated most recently by Shirai(8). 

♦ 

4. Waltz—The extent to which completeness of description 
can replace flexibility of control (9). 

5. Charniak—The role and extent of knowledge in 




Frontal 


understanding (1). 

6. Winograd—Semantic/syntactic integration, 

attack on global problem; large system structure. 
Visual/linguistic analogies (10). 

7. Hewitt—The power of a problem oriented control 
structure. The ease cf a pattern directed data base. Two way 
flow of control; flexibility of non destructive failure (3). 

As a framework for these developments, we have the Vision 
System. This has provided the experience and facilities for a 
broad based background in vision research plus the specific 
technical and organizational rsults of the System, itself 
(Complemented by the more formal seminar structure). 

Perhaps most importantly we want to deal specifically with 
the problem suggested in the first paragraph. This is very 
different from most previous efforts at visual recognition. 

Often the descriptive problem has teen dealt with separately 
as a pattern recognition issue. When real visual input has been 
used, the aim has been generally to transform this input, with a 
single low level predicate, into a "preprocessed" data base. 
Features could then be extracted from this data base and dealt 
with as a pattern recognition or tree search problem. 

In fact, we maintain that there is no real problem, on the 
macroscopic level, in distinguishing a respectable set of real 
objects. In some respects, the more complex and unusual the 
objects the better. It should not be at all difficult to 
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"separate these objects in parameter space" 
redundantly distinguishing features. 

The problem is to get some of these gross features: 

a. from amid all this redundant mass cf information 

fc« in particular, from the mass of "meaningless" low level 

data 

c. in differing: 

1. individual samples of objects (variations in size, 
shape, orientation, texture, etc.) 

2. scene contexts (occlusions, shadows, etc.). 

Cn a local level these changing conditions create great 
differences. If we deal only from the "bottom up" we cannot hope 
to deal with such chaotic variation. 

However, with higher level interaction we can hope for 

global guidance at levels that can deal with such variation. The 

old saw: "it's easier to find something if you know what you're 

looking for". Vie need to know what we are looking for to find 
the features. 

I think my goals can be approached in two stages. 


1) First I will pick a single object of some semantic richness, 

e.g. a hammer. I will then write the world's greatest hammer 
recognizer. This is not as trivial as it may sound. 

The program will needless to say not merely reply "hammer" 
when faced with any object, on the grounds that hammer is the 
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only object it knows about. Tt will have to decide whether the 
object is a hammer cr net a banner. 

Tt should he able to make this decision about your 
roommate's favorite hammer (or screwdriver). It should he ahle 
to decide after a stranger is allowed to place the object in the 
field of view ard arrange the lights. It should decide correctly 
when visiting Cub Scouts core ir and are allowed to empty their 
pockets over and around the object to be studied. 

This is a hard problem. 

2) Assuming* Part I is net a thesis in itself, the next stage will 
be to integrate knowledge of several objects into a system. 
Hopefully the experience of part Iwill provide principlesfor 
encoding visual knowledge that will facilitate adding the initial 
ability to deal with several new objects. 

V/e will want to design the analytical structure to take 
advantage of the hypothesis and verification techniques developed 
in Part I. I.ote that the problem domain of I’art I encourages us 
to explore the practical implications of our philcsphical bias: 
application of specific higher level knowledge. 


Preceding page blank 
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Problem 

Machine Vision involves both "low level" image processing 
and "high level" descriptive analysis. The failure of Machine 
Vision to date has been in adequately linking these levels. 

Much of the work in this field has restricted itself rather 
strongly to one or the other of these levels. There are several 
reasons for this. In the beginning there was a tendency for 
image processing people to believe they could "do it all" with a 
clever enough set of Fourier Transforms. The descriptive 
analysts, on the other hand, tended to underestimate the problem 

of preprocessing a suitable data base for their descriptive 
schemes. 

Problems in each of these areas have been formidable enough 
to perhaps demand single-minded concentration. But now that a 
groundwork has been laid we are in a position to take a more 
comprehensive approach. It has in fact become clear that such an 
approach is mandatory for continued progress in Machine Vision 
(4-7,11-13). The pressures for thesis work at a single level of 
the problem have begun to lead away from the overall goal. 
Search for the most sensitive line finder sidesteps the natural 
context in which the problem is to decide which lines to ignore, 
not which obscure lines to find. Yet work continues on idealized 
"picture grammars" which assume the preprocessor has given them 
just the data base they need (without any knowledge of those 



Page 182 


needs). 

Part of the reason that work on these two aspects of the 
vision problem has often remained unnaturally isolated must be 
the distinction in interests and skills required for work in the 
two fields. We therefore think it might be advantageous for two 
students to pool their experience in these separate areas. In 
any event a "pincer" attack is called for specifically on the 
interface between high level and low level analysis. 

This does not mean regarding present accomplishments in 
these areas as ’’black boxes" and wiring them together. Rather, 
processes at each level must be organized with the requisite 
cooperation in mind. 

Seeing at any level beyond the feature point is a matter of 
organizing visual information according to some visual model of 
the world. In the past we arbitrarily used only the most 
elemental physical units of the model to process the visual data- 
-the most basic physical regularities and anomalies such as lines 
and points. The rest of the higher level knowledge was stored 
separately to be "matched" against data that could hopefully be 
organized into meaningful structures without knowing which 
structures were meaningful. This knowledge must also be part of 
the visual machinery that organizes the raw visual input and thus 
"sees". Instead this information has teen so independent of any 
inherently visual process that researchers have been able to 
point pridefully to their ability to perceive and program this 
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"recognition" apparatus as instances of highly general abstract 
mathematical models. 

In point of fact, higher level context assumptions (a 
parallelipiped environment, for example) have long been implicit, 
or even explicit, in much of our work. Experience has shown the 
utter necessity for such guidance in order to make sense out of a 
chaotic sensory reality* It is time to fully integrate our 
higher level processing into the organization of our visual data. 

V/e have seen how difficult it is to obtain even a line 
drawing of a cube without suspecting that we are looking at a 
cube. How can we expect to obtain some idealized data 
representation of a horse and then recognize that representation 
as a horse. We cannot choose the ideal descriptive model of a 
horse analytically and expect seme syntactic predicate to produce 
this description for us. 

The conclusion we come to is that there is a need to 
eliminate the old distinctions between preprocessing, description 
and recognition. One cannot obtain a preprocessed result which 
is then analyzed to produce a description which is then matched 
with a model to be recognized. Rather seeing is a homogenized 
process, the "model” is built into the structure of the 
processor, the description of a data set is the protocol of its 
processing. Recognition is coincident with description. 

Recognition is not a matching of templates against a 
processed data structure. Rather it is a many layered silk 




screen process that begins with the raw input and leads to the 
richly patterned perceptual world we seek. 
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Issues 

V/e feel that an approach which is predicated on the intimate 
interaction of descriptive apparatus and description predicates 
is the best hcpe for practical progress in Machine Vision. 
Beyond that we believe that such a project would provide an 
appropriate laboratory for the illumination of several current 
A.I. conceptual concerns. 

1. Heterarchy 

Problems in reconciling low level results with high level 
idealizations, even in a very simplified domain, provided one of 
the early motivations for the heterarchy concept (4-7,12). 
Return of recognition level advice to the preprocessor is perhaps 
the most salient example of heterarchy practice (4). It is 
significant that, except on a most general level, even this case 
study of heterarchy has not been implemented and its implications 
certainly not fully explored. 

Our investigation of heterarchy in vision would emphasize 
the crucial role of recognition level knowledge. 

We do not analyze heterarchy in terms of interaction or 
advice between major discrete processing modules or stages. We 
do not view "high level" knowledge as criticizing the descriptive 
structure resulting from low level "preprocessing". Rather we 
believe that in general no useful structure can be derived 
without high level knowledge directly involved in the 
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construction process from the "beginning. 

In any case, a working fully ramified case study should 
provide useful insight into the theory and practice of 
heterarchical interaction. In particular: 

2. Sensory/perceptual systems 

The heterarchical interaction between lew level sensory 
systems and high level perceptual systems is of particular 
interest to A.I. research at present. Having worked "down" from 
chess-playing and integration, A.I. is now facing the far more 
obscure problem of enabling machines to duplicate the essential 
"automatic" processing of real world sensory data. Techniques 
and processes abstracted from our results in melding low level 
and high level visual processing should prove relevant when we 
provide our robot with other sensory apparatus. The possibiity 
of higher level semantic intervention in the auditory analysis of 
speech, for example, is already a live issue. 

3* Knowledge—"the size of infinity" 

In developing a system that can deal with in a real world 
context we will face certain problems analogous to those faced by 
Charniak in his work on understanding childrens' stories. 
Previous successes in machine vision, or even machine pseudo- 
vision, i.e. analyses of hypothetical input, have dealt largely 
with a severely restricted domain. The recognition set has been 
either highly stylized or simply very small. There is an analogy 
here to syntactic uni-directional theorem-proving systems, which 
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break down on a non-restrieted data base. First we will have to 
get a practical idea of what the size of infinity is in the 
visual domain, and then we will need techniques for organizing 
it. In particular: 

4. Knowledge—as procedures 

Our knowledge—about shapes as well as subjects—will be 
organized as procedures. This is more than a fashionable device 
in a system where knowledge will often consist of knowing the 
right questions to ask of some other module, preprocessing 
modules in particular. ("Preprocessing" is obviously a misnomer 
here, reflecting an organizational mode we hope to replace.) 

Recognition will not consist of preparing a description of 
an input scene and matching it with a description of a model. 
Rather description and recognition will occur simultaneously 
during the processing of the scene. A description will be 
essentially the protocol of a process, not a result. 

5. Parallel processing 

If parallel processing techniques are to prove significant, 
this would certainly seem to be an area in which they might 
demonstrate their value. V/e envision a system in which 
processing would proceed simultaneously on visual subunits or on 
conceptual subprocesses. Proceedings would be dependent 
continually on the results of intermodule communications and 
questions, both as to the results being attained in different 
"higher" level investigations, and the results of additional low 
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level processing. It would seem plausible that dialogues of the 
following style could take place profitably: 

a: I think these four things are legs. Is anybody looking 
at anything above them all? 

b: Yes, I am. 

a: Can you tell me if it's planar or bumpy? 

b: Not yet. 

a: Well, look, work on it; take your time and wake us when 
you have something. 

a: (Yawn) Wazzat? Bumps huh? Probably an animal. We'll 

activate hoof, paw and knee searches. You send your stuff to an 
animal tody parser. 

c: I can't find any subdivisions on this thing for head or 
tail. Can anybody make head or tail out of any nearby blobs of 
relative size x and position n? 


This particular dialogue may be vacuous. Investigation of many 
possible dialogues may reveal some essential usefulness of this 
type of organization. 

6. Grouping 

As even the brief dialogue above indicated, grouping 
problems will be central. Guzman-like techniques will not 
suffice, particularly for higher level organization. Rather than 
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very sensitive local 


predicate? capable of distinguishing fire 


variations, we will require broader based cruder predicates 


capable 


of determining relative homogeneity. Contextual, 
conceptual cues will be needed to distinguish overlapping objects 


froii functional groups. A program that knows about collars will 
not be dismayed to find a dog's head separated from its bodv r ; it 
v/ill be capable of seeking the "severed head". Alternative 
hypotheses will be explored for conjoining parts of the picture. 


ignoring occlusions, or melding sequences into single units. The 
same subscene may be viewed in each of these ways: as linked 
units, overlapping distinct units, single textured units. 
Successful interpretation will direct the choice. 

7. Organization 

Direction will be available from both the lov/er level and 
the upper level. lhat is, knowledge about a shape as v/ell as an 
object or class of objects, will be embedded in procedures which 
will in turn direct the flow of processing through other 
procedures. This flow will not be locked into any piece—by—piece 
level—by—level decision tree or network search. The system 
can skip to high level hypotheses, plan an approach on that 


basis, fail, review what it found in the process and use this as 
a basis for some more "from the ground up" investigation. 
Details can be verified on a hoof before during or after the 
search for a head; whatever is expedient. 



Approach 


We expect our progress or this project to reflect the 
structure of the results. That is, both will proceed by a 
process of "successive approximation". 

Professor Papert has characterized one approach to problem 
solving as "neglect all complication, try something". The 
results may serve as a "plan" on which to base further 
refinements or a basis for "debugging". 

There is a distinction between simplifying the problem and 
simplifying the solution. One technique involves splitting the 
problem into a number of separate or successive simplified 
problems. For example: ignore shadows, texture,etc., and just 
consider ideally lighted ideal planes, then consider shadows, 
then consider texture, but no shadows. Etc. This approach can be 
very fruitful. However, it may leave us far from a solution to 
our total problem, particularly if the various aspects of the 
problem are related in a non-linear fashion. In this case it may 
prove necessary at times to approach the total problem with a 
simplified solution. We can then rework this solution 
successively to approach a more adequate solution. The 

intermediate results serve to indicate directions for 

improvement. We may "project" structural aspects of the problem 
from the solution, rather than relying on an a priori division of 
the problem. 
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We expect a continuing dialogue durirg the research that 

reflects and suggests the dialogue that will be built into the 
system: 

>Is that side straight? 

>1 can't tell; it peters cut in spots. 

>0h really? Maybe they're lined up wrinkles. Are there 
more of them? 

>Could be. 

>Can you characterize their profile? 

>Yes, I can give you a "possible wrinkle" predicate; but 
that predicate will also work on soft corners. 

>But the surrounding planes for a wrinkle will be similar? 

>Yes, and aligned shadows often occur with wrinkles. 

>Now can you ... 

>No but I can ... 

>Well then if I tell you ...» then could you ... 

>... 

The important thing is that many of the technical realities 
of the system will flow from real experience with the problems of 
processing real data. This does not mean that the results will 
lack theoretical content, but that the theory will bear a valid 
relationship to the reality of the vision problem. We want the 
theory to flow from and reflect the structure of visual 
experience, rather than attempt to distort visual data to conform 
to preconceived and inappropriate analytic theories. 




Benchmarks 


We might proceed in several phases. A possible sequence 
would he as follows. 

As a first stage exercise, we could consider the development 
of b system that dealt with simple geometric models, e.g. cube, 
cylinder. This system would provide experience in melding high 
level knowledge with a suitably varied set of low level 
predicates. The idea is not to push any one predicate or 
approach to its limit but to allow the extent of higher level 
knowledge arid variety of low level approaches to work back and 
forth, to zero in on the correct perception. 

A brief example is a simple cube. We avoid working very 
hard to find the precise edges from a feature point analysis. 
Rather we obtain some rough regions with a crude homogeneity 
predicate (that averages out not only surface noise but 
textures). We massage these a bit and get a rough idea of their 
shape, if we suspect a parallelipiped, we may apply a finer test 
to verify this shape. We find enough to guess a cube or at least 
a planar object. V/e then apply a crude line verifier over the 
wide band between regions. Meeting success we may home in on the 
lines with a sensitive line verifier in a narrow region. (If 
some regions were lumped into one at the start we use our models 
to guide predicates in a parsing attempt (4).) 

This example already illustrates several interesting points. 
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Unlike a pass criented system, we do rot have to apply all our 
precicates at once, tut only as (and where) needed. We make use 


of troad based, region oriented predicates to avoid the problems 
of high sensitivity until we have some hypotheses to puide the 
search for finer features. By workinp down from the peneral to 
the specific we avoid lcsinp the forest for the weeds. And even 


at the lov/est level we are still dealinp with a broad based 
predicate, a lire verifier. We also avoid an initial commitment 
to a world model of straight lire planar objects. 


The next stage would provide experience in dealing with a 
complex of surfaces in non linear terms. A limited set of "real” 
objects, a set cf tools for example, wculd provide a miniworld. 
The second stage system would be able to function in this world. 


Here we would have to incorporate a greater variety of data types 
and predicates cr procedures. 


Another simple example is a hammer. We find an initial 
region. We suspect and verify a roughly rectangular shape. 
Relative length versus width prompts a handle hypothesis (or vice 
versa). This initiates searches for regions at either end, 
either crosswise or colinear to the handle axis (we could have a 
screwdriver). V/e find a chunky region crosswise at one end. We 
hypothesize s hammer with handle and hea.d. We move back down to 
verify a few fine details to assure our success. 

hotice here some further principles emerging. Precisely 
what is required to see a hammer head varies depending on where 



or from what direction you "enter" the observation. Think, for 
example, how carefully you would have to draw a hammer head to 
make it identifiable in the following contexts: at the end of a 
roughly drawn hammer handle, standing alone by itself, at the end 
of a carefully drawn hammer handle, in a picture of the head 
alone to be viewed only partially and in isolated pieces. The 
requirements are different if we move "down" from a knowledge, or 
hypothesis, of "something at the end of a hammer", or if we move 
"up" from a detailed picture of the contour of a piece of the 
head. Generally we have a range of redundant information that 
specifies an object; any particular set of details, e.g. of 
shape, are not always required for identification, and may be 
easier to verify than to obtain by "preprocessing". 

A flexibly programmed "understanding" of hammer heads should 
be able to be "entered" at several levels. Processing should 
proceed in parallel, with mutual calls, modified by the current 
state of knowledge. 

It may be appropriate here to initiate a Charniak-type study 
of "everything we need to know" about, e.g. hammers, in order to 
perceive them in varied contexts. Why should such extensive 
knowledge not be necessary to "understand" in visual contexts, 
just as it is required in verbal contexts? 

Another miniworld would be established to provide experience 
in dealing with classes of objects. A set of saws, for example, 
or a set of doll house furniture, could form such a class. The 
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third stage effort would provide a system that could move easily 
through the perceptual space of this class. 

Some of Winograd's ideas on organizing knowledge might he 
useful here. A systemic grammar might be a useful metaphor for 
at least one aspect of the organization. Also some convenient 
method of inputting knowledge would be useful at this point. 
Ideally the system would be such that someone could eventually 
hook up the output of a Winston learning program to the input of 

this system. (And the output of this system to the input of a 
Winston learning program, of course.) 

Finally, a rich miniworld would be chosen to combine our 

previous experience in a more varied environment• A cutaway doll 

house, or a multipurpose workbench, for example. By the 

completion of this last stage our general principles for an 

integrated perceptual system should have been demonstrated, and 

new principles for implementing this integration should have 
emerged. 

This is rather an ambitious program. We could only hope to 
lay the groundwork really, to build an instructive testimonial to 
the possibilities derivable from a proper conjunction of high and 
low level knowledge. 


% 



Summary 


In summary, in order to make a quantum jump in vision 
research, a number of bold steps are required. The thinking of 
pre-eminent theoreticians in the field has long tried to push us 

in these directions. However, "practical" considerations have 
too long held us back. 

Instead of studying the parts and then deriving the "glue", 
we must study the glue and then derive the parts. 

It has for some time now been recognized that the real 
problems in vision lie in understanding the cooperation of the 
various subprocesses. In practice, it has been easier to define 
and deal with specific pieces of the spectrum of visual 
knowledge. However, we eventually end up sitting around 
bemoaning the difficulty of putting the pieces together. 

The solution is not simply to learn more about more pieces. 
Neither is it to treat the pieces we have as "black boxes" to be 
tied together. Aside from the theoretical arguments that could 
be mounted against this approach, it has proven rather barren so 
far in practice. To learn something about the mysterious 
"binding energies" as it were, we must simply grit our teeth and 
attack the problem directly. This approach will often mean 
messy, tentative, ad hoc progress. V/e must be willing to pay 
that price. We may utilize our hard won piecemeal knowledge, of 

However, when we have finally won our way to some 


course. 
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understanding of the interactive process cf vision, we may find 

ourselves in a letter position to identify the essential 
subprocesses involved. 

Instead of generating data types and predicates to handle an 
idealized data base we must generate idealized data types and 
predicates to handle a real data. base. 

Becoming absolute experts on line drawings will only qualify 
us as experts in graph theory. The humor of this approach is 
evident when we consider that "real" physical data is 
manufactured at great cost and effort to match this idealized 
data base, and even then the predicates derived for the line 
drawing model cannot succeed in producing a satisfactory 
translation of the physical objects into line drawing data types. 

Our line drawing studies have provided us with some useful 
methodologies and results that should provide guidance and 
submodules for a mere general study. However it is time to 
return for inspiration and guidance to more realistic data: not 
to set up another panacea predicate, but to extract whatever 
information is required to organize the sensory input; not to 
organize simply and dogmatically as line or corners, but as 
recognizable perceptions, as is appropriate for the data. 

Most fundamentally, the easy but artificial distinction 
between seeing, describing, and recognizing must be broken down. 
We cannot merely organize the visual input according to some 
"neutral" data base. We must not partition out part of our 



knowledge to function as a template to "match" our results. The 
only way we can hope to organize the visual data to "match" a 
high level form is to use that high level knowledge to perform 
the organization. The concept of a "model" as such is outdated. 
The medium is the message. Recognition is the process of 
description. Description is the process of recognition. 
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A HETERARCHICAL PROGRAM ECR RECOGNITION OF POLYHEERA 


"by Yoshieki SKIRAI 

ABSTRACT 

Recognition of polyhedra "by a heterarchical program is 
presented. The program is based on the strategy of recognizing 
objects step by step, at each time making use of the previous 
results. At each stage, the most obvious and simple assumption 
is made and the assumption is tested. To find a line segment, a 
range of search is proposed. Cnee a line segment is found, more 
of the line is determined by tracking along it. Whenever a new 
fact is found, the program tries to reinterpret the scene taking 
the obtained information into consideration. Results of the 
experiment using an image dissector are satisfactory for scenes 
containing a few blocks and wedges. Some limitations of the 
present program and proposals for future developments are 
described. 
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1. INTRODUCTION 

We do not know how to make a program to recognize objects 
visually as well as a human being. One of the shortcomings of 
many computer programs is, as Minsky has pointed out, their 
hierarchical structure. A human may recognize objects in the 
context of the environment. The environment may be recognized 
based on his a priori knowledge. The recognition procedure is, 
however, well programmed so that the simple obvious parts are 
recognized first and the recognition proceeds to the more 
complicated details based on the previous results. 

The work in this paper studies an example of a heterarchical 
program to recognize « polyhedra with an image dissector. Most 
previous works begin by trying to find feature points in a entire 
scene and make a complete line drawing. It is very difficult to 
get a complete line drawing without knowledge about a scene. If 
the line drawing has some errors, the recognition by a theory 
based on the assumption of the complete line drawing such as 
Guzman's might make still more serious mistakes. Our work is 
an attempt to recognize objects step by step, at each time making 
use of the previous results. 

We assume in this paper that the difference in brightness 
between objects and the background is large enough to detect the 
boundary approximately. At present, this program works for 
recognizing moderately complicated configurations of blocks and 
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. .. . 

wedges. The limitations and proposals for future development 
are described later. 








i- 
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2. GENERAL STRATEGY 

2.1 Priority Of Processing 

Por convenience, we set up the edges of the objects in a 
scene as falling into 3 classes. A line formed at the boundary 
between the bodies and the outer background is a contour line of 
the bodies. In Fig.1, lines AB, BC, CD, DE, EF, FG, GH, HI, IJ, 
JK, KL, LM, MN, NO, AO, VW, VJX, XY, YZ and ZV are contour lines. 
A boundary line is a line on the border of an object. Contour 
lines are boundary lines. In Fig.1, the boundary lines are the 

contour lines and lines on the boundary between two bodies, i.e. 
CP, PH, IQ, QR, and RM. An internal line occurs at the 
intersection of two planes of the same body. Lines JS, LS, QS, 
PT, NT, AT, FU, GU, DU and XV are internal lines. 

The global strategy is shown in Fig.2. At first, the 
contour lines are extracted (because we assume a priori enough 
contrast between the objects and the background). If more than 
one contour is found, as in Fig.1, one contour for bodies B1, B2, 
B3 and another for body B4, then the boundary lines and internal 
lines are searched one by one for each contour. The global 
strategy in block 2 in Fig.2 is as follows. 

A) Find boundary lines before finding internal lines because 
boundary lines often give good cues to guess internal lines. 
Note that to find boundary lines implies to find bodies. 

B) In searching for lines, different situations require 
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examination of larger or smaller areas. In our strategy, the 
smaller the area required to search lines, the higher priority 
we give to that search. In Fig.1, for instance, to determine 
the existence of a extension of line EC, it is enough to search 
a small area whose center is on the extension of the line. To 
find line IC, however, we should consider all possible 
directions of a line between IP and IJ. Thus the former search 
has priority over the latter. 

The priority to extract the most obvious information first 
is in the following order. 

1. If two boundary lines make a concave point (such as 
point B in Fig.3), try to find an extension of them. If only 
one extension is found, track along this line. Most of such 
cases are like in Fig.3 (b) where one body hides the other. 
V/e can determine to which side of body this line belongs. 

2* If no extensions of two concave lines are found, try 
to find another line which starts from the concave point. If 
only one line is found, track along this line. Most of these 
cases are as in Fig.3 (c) where it is not clear locally to 

which body this line belongs. In Fig.3 (c), line ED belongs 
to the upper body# but this is not always true. That is the 
lines AB, BC, BD are not sufficient to decide the relation. 

3» If both extensions of two lines are found at a 
concave point, try to find a third one. If only one line is 
found, track along this line. This is the case as shown in 
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Fig.3 (d) where the third line is the boundary line. 

Whenever tracking terminates, an attempt is always made to 
connect the new line to the other lines that were already found. 
If more than one line segment is found in (1), (2) or (3), the 
tracking of those lines is put off hopefully to be clarified by 
the results of knowledge obtained in simpler cases. Fig.4 
illustrates two extensions found at concave point P. The 
interpretation of the two lines is put off to treat simpler cases 
first. That is, one would continue examining the contour and 
lines AE and CD might be found next; then, by a circular search 
at points E (which is explained later), line EP would be found. 
At this stage it is easier to interpret lines AB and EP as 
boundary lines which separate two bodies. Then line DP might be 
found similarly and interpreted correctly. 

4. If an end of a boundary line is left unconnected as 
PQ in Fig.5, try to find the line starting from the end point 
(Q in this example) by circular search. If multiple lines 
are found, try to decide which line is the boundary. If a 
boundary line is determined, track along it. In Fig.5, the 
dotted lines are found by circular search and the arrows show 
the boundary lines to be tracked. 

5. If no line is found in the case (4) as stated above, 
extend the line (PQ in this example) by a certain length and 
test if the line is connected to other lines. If not, then 
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apply circular search again as in (4). This is necessary 
because the termination point of the tracking is not always 
precise. 

Note that this process can "be repeated until successful 
(that is either the line is connected to other lines or line 
segments are found by circular search). 

6. If the boundary lines of a body are known, select the 
vertices of the boundary that might have internal lines 
starting from them. The selection of vertices is based on 
heuristics such as selecting upper right vertex rather than 
lower right vertex. At each vertex, try to find an internal 
line which is nearly parallel to other boundary lines. If 
one line is found, track along it. In Fig.1, for example, 
internal line JS is parallel to the boundary line KL or 10, 
and QS is parallel to R1 or IJ. Line FU is parallel to FD 
and XV is parallel to XZ. Thus it is often useful to find 
internal lines parallel to boundary lines of the same body. 
Note that search for parallels has small area. 

7* If no line is found in (6), try to find one by 
circular search between adjacent boundary lines. When one 
line is found, track along it. In Fig.6, circular search 
between BA and BC is necessary to find the internal line BE. 

8. If two internal lines meet at a vertex, try to find 
another internal line starting at the vertex. This process 
is used in two cases. One is where no internal line was 
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found in (7) “because of little difference in “brightness 
between adjacent faces. Suppose in Fig.1, that the internal 
line SJ was not found at vertex J, but that LS and QS were 
found. Then try to find an internal line starting at S 
toward J. If there is enough contrast near S, a line segment 
is found. The other case is where a body is partly hidden by 
other bodies. In Fig.6, the triangular prism is partly 
hidden. After BE and CE are found, EF is searched for. In 
both cases, the direction of the line is sometimes 
predictable and sometimes not. If it is predictable, then 
try that direction. If it is unpredictable or if the 
predicted direction failed, then apply circular search 
between the two internallines. If one line is found,track 
along it. 

9. If an end of an internal line is not connected to any 
line, try to find lines starting from the end by circular 
search. If lines are found, track along them one by one. 

10. If no line is found in (9), extend the line by a 
certain length as in (5) and test if it is connected to other 
lines. If not connected, try circular search again as in 
(9). This process can also be repeated until successful. 
Fig.7 illustrates this process. In Fig.7 (a), line MN' is 
not connected to others at N% thus step (9) is tried at N' 
and fails. The line is extended to P and (9) is again 
applied. This process is repeated until the line is 







connected to line KL at N. Fig. 7 (b) shows that line HI is 
extended by this process tc P where a new line is found "by 
circular search. Similarly line CG is extended to F . This 
process is useful so as to not miss a new tody sitting on an 
obscure edge. 

At each stage when an above step is finished, the obtained 
information is interpreted as shown in Fig.2 (block 3). For 
instance, if tracking along a line terminates, a test is made 
whether the line is an extension of other lines and/or the line 
is connected to other lines at a vertex. If a boundary line is 
connected to another boundary line, the body having the lines is 
split into two bodies and the properties of both lines and 
vertices are stored in an appropriate structure. In Fig.8, for 
example, line N'P' is obtained by tracking starting at point N. 
This line is interpreted as an extension of HU, and HN and N'P' 
are merged into one straight line using the equations of these 
two lines. Then, it is connected to CO and Fig.8 (b) is 
obtained. Before the line was connected to CC, there were two 
bodies El and B2 as in Fig.8 (a). Now body B2 is split into two 
bodies B2 and B3. We can interpret line NO as the boundary of P3 
which hides a part of B2. The other properties of lines and 
vertices are obtained similarly at this stage. 
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2.2 Example 

We illustrate the entire line-finding procedure with the aid 
of the example shown in Fig.9. At first, the contour lines AE, 
BC, CD, IT, FF, FC, OH, HI, IJ, JK and KA are obtained as shown 
in Fig.9 (a). Step (1) described in the previous section is 
tried for the concave points G and J. In this example, the 
position of G is not precise enough to find the extension of FG. 
On the other hand, a line segment is found as an extension of the 
line KJ. KJ is extended by tracking as far as L* Because there 
is no other point to which step (1) is applicable, step (2) is 
tried for point G. One line segment is found and extended till 
tracking terminates. Thus a line G'M' is obtained as in Fig.9 

(b). This line is interpreted as an extension of FG and 
connected to JL. Then the position of point F, G, L are adjusted 
to as shown in Fig.9 (c). Now two bodies B1 and E2 are created 

by the boundary lines GL and JL. It is important to notice this, 

for it means that step (1) is again applicable (to point L) at 
this stage. Thus line FL is extended as far as M in Fig.9 (d) 
(Note that line MN # has not yet been found). LM is interpreted 
as an extension of FL but the end point M is not connected to any 
other lines. Thus vertices F, G, L and end point M are adjusted 
considering the new line LM. Here neither step (1), (2) nor (3) 

is applicable, so that (4) is now applied to M. Three lines are 

found by circular search as Fig.9 (d). MN' is determined as a 
boundary line and extended by tracking. When it terminates, the 
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line is connected to boundary line BC at vertex N as in Fig. 9 
(e). Body B1 splits into body B1 And B3. It is known at this 
stage that El is hidden by B3 and B2 is hidden partly by F3 and 
partly by B1. Next, step (6) is applied to each body one by one 
at each time selecting the easiest body for proposing the 
internal lines (in this example, the order is B3, B1, B2 because 
P3 hides B1 which hides B2). Internal lines CO and MO are found 
and connected at vertex 0, but no line segment is found using 
step ( 6 ) and (7) applied to vertex E (this stage is shown in 
Fig.9 (e)). Step (8) is applied to vertex 0 and a line segment 
toward F is found. This is extended by tracking as far as E' as 
in Fig.9 (f). Line OE' fails to be connected to any other lines 
which activates step (10). After a few trials, OF' is extended 
to connect to vertex E. Similarly, internal line AM is obtained 
for body B1 and line IP is obtained for B2. When every step has 
finished, three bodies are known together with the relationships 
between them. 









3. ALGORITHMS 


This section describes the details of the algorithms that 
are used in finding contours and in the steps stated in section 
2. Some of them, such as tracking and circular search are used 
in more than one step. An algorithm used in more than one step 
may be slightly different in each step but its essential part is 
not changed. In the tracking algorithm, for example, some 
changes occur depending on whether tracking is used for boundary 
lines or internal lines. 

3.1 Contour Finding 

Fig.10 shows the outline of the procedure to fird contour 
lines. The picture data obtained with an image dissector 
usually consists of a large number of points (say about 100,000) 
each of which represents light intensity level. To speed up the 
processing, one point for every 8x8 points is sampled. This 
compressed picture data consists of 1/64 the number of points in 
the original picture. To find the contour, this data is scanned 
till a contour point is found. The judgement of contour point is 
based upon the simple assumption that there is enough contrast 
between the background and objects. It is then checked whether 
or not the point is a noise point. If it is a real contour 
point, trace along the contour. Thus a set of contour points are 
found. Then, the picture data is again scanned until a new 



contour point is found. This process is repeated for all the 
picture data. When all the sets of contour points have been 
found, each set is separately analysed. 

Suppose a certain set of contour points is to be analysed. 
We now return to the original high-resolution picture. We can 
guess approximately the position of the boundary point in the 
original picture data which corresponds to the first point found 
in the sampled picture. The precise boundary point is searched 
for near this point. A set of contour points is obtained by 
tracing from this point in the same way as in the sample picture. 
A polygon is formed after we connect contour points one by one. 
To classify the points of this 'curve' into segments,, the 
'curvature' of the polygon is used. This curvature in a digital 
picture is defined here for convenience as shown in Fig.11. Each 
cell in the figure represents a contour point. The curvature of 
a point P is defined to be the difference in angle between PR and 
PQ (d), where Q and R are a constant number of points away from P 
(6 points in this case). If we plot the curvature along the 
contour as shown in Fig.12, we can tell what part is near a 
vertex and what part lies in a straight line. 

Note that curvature is not very sensitive to noise or 
digitization error. If we integrate the curvature in part of a 
straight line, the result is nearly zero despite the effect of 
noise. If we sum up the curvature of consecutive points whose 
absolute value is greater than some threshold, we can determine 
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Fig. 12. Example of curvatui 
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Lnition of curvature 
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the existence of a vertex* That is if this sum of the curvature 
of such points exceeds a certain threshold, there is a vertex 
near those points. Thus every contour point is classified to he 
either in the straight part of a line or near a vertex. Using 
points which belong to the straight part of a line, the equation 
of the line is calculated. Then each vertex is decided as an 
intersection of two adjacent lines. 

3.2 Line segment Detection 

A line Segment is detected given its direction and starting 
point. This procedure is used in most of the -step® stated in 
section 1. The procedure consists of two parts. One is to 
detect the possible feature points which are to be regarded as 
elements of the line. The other is to test whether or not 
obtained feature points make a line segment. 

In detecting feature points, we should consider various 
types of edges. Herskovits and Einford classified the light 
intensity profiles across an edge into 3 types, namely step, roof 
and edge—effect, and proposed 3 types of boundary detectors. In 
this paper, a roof type detector is not considered because roof 
- type edges can be detected by a step detector or an edge-effect 
detector. In addition, most roof type edges are accompanied by 
step or edge-effect types. We set up local Cartesian coordinates 
U-V such that U is the direction of the line segment to be 
detected. Let I(u,v) denote the light intensity at point (u,v). 




Page 221 


and define the contrast function F (v) at (u,v) as 

F (v) Jz (l(u + i,v + j)-I(u + i,v - 3 )} 

j =1 i=-i m l ; 

Suppose we have an intensity profile as shown in Fig.13 (a), 
F u (v) at P in Fig*13 (b) is the difference of summed intensity 
between area A z and A,. F H (v) for a typical step type profile 
(Fig.13 (a)) is shown in Fig.13 (c) in which the edge is detected 
as the peak. The typical profiles of F^v) for other types are 
shown in Fig.14 where the edge is detected as the middle point 
between positive and negative peaks. 

The basic procedure, therefore, is to detect the peak of 
F u (v) and its position. The necessary properties for a peak are 
as follows (see Fig.15). 

(a) If F u (v) ranges from v* to v r , there must exist the 
maximum of F Ui (v) at v m other than v* or v f . 

(b) F Vb (v m ) > fm where fm is threshold. 

(c) There must exist a minimum of F^v) at v, between v* and v m 
and a minimum of F u (v) at v^ between v m and v r such that 
*V(v*) - F*(v,) > f d 

F^(v m ) - Fvi(v x ) > fd where f d is a threshold 
If such v^ is found, the left of the peak (v^) and the right 
of the peak (v^) are determined as the intersection of F u (v) with 
the line F u (v) = f^ as shown in Fig.15. The value of f^ depends 
on F^(v^) and is represented as 

f t = c i F u.( v tn) + c o 
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where c y and c 0 are constant and 0 < c, < 1, c 0 > 0 
The position of the peak Vq is obtained as the middle of v. and 
V H* more than one peak is found between v* and v r , the 

v o which is nearest to the middle of V£ and is adopted. 
A negative peak is similarly detected. A feature point for an 
edge-effect or roof is obtained as the middle of the positive and 
negative peaks, if both are found, (although the threshold f„ is 

FV) 

not the same as in the simple positive or negative peak 
detection). This method for the detection of peaks and 
positions is not appreciably affected by noise. 

The other part of the line segment detection is a test of 
the co-linearity for the detected feature points. Suppose 
concave boundary lines L 0 and Lj meet at Pq and suppose the line 
segment extending L 0 is tested as shown in Fig.16. Feature 
points are detected in a rectangular search area with given 
length and width whose direction is equal to that of L 0 (= U), at 
an appropriate place where the detection of feature points is not 
affected by the edge corresponding to . Feature points are 
detected along the direction v at the center points P f ,F t ,... f F rt 
sequentially. If positive peaks are found at M, ,M 1 ,...,M n as 

shown in the figure, the linearity of the points are tested as 
follows. 

(a) The number of the feature points must exceed a threshold 
number n . 

(b) The deviation <5 of the points in line fitting with the 
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least square method should he less than a threshold * 

(c) let U' denote the direction of line segment obtained ty 
line fitting, 

Ju' - u| < u d 

where |U" - U| denotes the diffrence in directions U' and U 

Similar tests are made for the different types of feature 
points. If more than one type of line segment is found, the 
selection depends on the following criteria. 

(a) If an edge-effect type is found, then it is selected. 

(b) For the line segment with 6> and U', let the criterion 
function C be 

C -O + wjJU' - U| where w is a constant 
The line segment selected is the one with smaller C. 

3.3 Circular Search 

Circular search is used to search for lines starting at a 
given point. The direction of the lines to be searched for is 
not known. The range of directions in circular search depends 
upon the particular case. Suppose two known lines L, and meet 
at P as in Fig.17 (a) and suppose we wish to search for lines 
lying between them. The search range <k is between two lines 
and whose directions are slightly inside of and Ij. 

respectively. If lines starting at point P of line L 0 are 
searched for as in Fig.17 (b), and L£ are similarly set inside 
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of L q . The center point P of the circular search is not always 
precisely determined, especially when tracking along a line has 
terminated at point P as shown in Fig.17 (b). Therefore circular 

search should not be too sensitive to the position of the center 
point. 

It might be natural to try to detect feature points, as 
defined in section 3-2, based upon F^Cv) along arcs around the 
center. The difficulty with this search is the classification 
of feature points into line segments if there is more than one as 
shown in Fig.18. To avoid this difficulty, a simple algorithm is 
used in this paper. Its basic method is to apply line segment 
detection successively in various directions. This is 
illustrated in Fig. 19 , where successive line segment detections 
toward u,,u ^ and u 3 are applied. The step of direction change 
and search area (A|,Ax and A 3 in the figure) are determined so 
that line segments of any direction near the center point can be 
found. Thus successive circular search along a line as shown in 
Fig.7 can find lines . starting at points between two adjacent 
center points (e.g. line L in Fig.20 starting between P, and p» ). 
The algorithm for line segment detection is the same as described 
in 3-2 except with respect to thresholds and search area. 
Because the search areas for different directions overlap each 
other, the same line segment may be found in different searches. 
Each time a line segment is found by line segment detection, a 
check should be made whether or not it is the same as the one 
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% represents a feature point 


Fig, 18, Illustrates difficulty in classification 

of feature points. 



Fig. 19. Successive line detection in circular search. 
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obtained by the previous detection. 

If the center point of circular search has not been 
determined precisely, it is not always possible to find all the 
lines starting at the given point. In Fig.21, for example, line 
Lx might be missed in circular search at P c . To avoid this 
inconvenience, when line segments are found (such as L, and Lj in 
the figure), a new center point P, is calculated based on the 
known line (L 0 ) and the obtained line segments (L, and Lj). 
Then circular search is applied again at P. , 

3«4 Tracking 

Tracking is used when a line segment is • given, to track 
along it until it terminates. The requirements for a tracking 
procedure are 1) the line should not be lost due to the effect of 
other lines or noise, and 2) the procedure should terminate as 
precisely as possible at the end of the line. These requirements 
are contradictory in that the termination condition should be 
strict to satisfy the second requirement which makes it difficult 

to satisfy the first. . The following algorithm is a compromise 
between these requirements. 

The basic procedure is to predict the location of a feature 
point and to search for it near the point using line segment 

detection. The result of the search is classified into the 
following 4 cases. 

(a) there is no feature point. 
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(b) a feature point is on the line. 

(c) A feature point is not on the line. 

(d) It is not clear whether or not a feature point is on the line. 

In case (a), the detection of a feature point is similar to 
line segment detection except that the type of edge is already 
known so that the thresholds stated in 3*2 can be adjusted based 
on the average peak of F u (v). The decision between cases (b), 

(c) and (d) is made using the distance d between the point and 

the line. That is 

If d < d, then case (b) 

If d > d£ then case (c) 

If d, < d d x then case (d) 

The threshold d, changes depending on the state of tracking. 

The state of tracking is represented by two integers m, and m^. 
which are set initially to 0. The value of m and m are changed 
for each case (a), (b), (c) and (d) as follows. 

(a) = m, +1 

(b) If ni| > i& 2 , m t = 11 , - 1 , (where m, is a constant) 

Otherwise, = 0, iij. = 0, and classify those feature points 

into (b) which have been clasified into case (d) 
in the previous steps of tracking. Adjust the 
equation of the line with these points and the 
present feature point 

(c) If d <. dj and m, > m a , m t = m, - 


1 
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(where d is a constant) 

Otherwise no change 

(d) m 2 _= m^ + 1 * and if m,> m^, m,= m,- 1 

The threshold d, is represented as 
d, = d 0 + w m m- 2 . (where d c and w m are constants) 

This procedure is repeated and tracking proceeds step by step 
extending the line until the termination condition is satisfied. 
The termination condition of tracking is either 
m. > or i, + mi > mt (where m n and m* are constants) 

The terminal point is defined as the last point classified into 
case (b). Fig.22 illustrates how this algorithm works. In 
Fig.22 (a), two lines cross at P c . Tracking might finish at some 
point beyond P© (P m in the figure) which satisfies the 
termination condition. The terminal point of tracking is, 
however^ determined more precisely near P©(Pj or P 2 .). In Fig. 22 
(b), P, ,P 1 ,p 3 ,P H are classified into case (d) increasing the value 
of m- 2 . which classifies Pj- into case (b). Then the line is 
adjusted with these points which are now classified into case (b) 
and tracking proceeds. 

Figs22 (c) and (d) illustrate that even if a part of the 
intensity profile is disturbed by noise or other lines, tracking 
does not terminate there. In Fig.22 (d), however, if the light 
intensity of the right side of L©changes across L^, the type of 
feature points might change across Lj • Thus feature points P^,Pt| 
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might not he obtained and tracking might terminate at P. . 

When tracking terminates, the line segment detection is applied 

at the extension of the line to see if another type of line 

segment is found. If found, ve adjust the line equation and 

tracking proceeds. If not found, tracking finally terminates at 

point P, and the position of P, is adjusted with the line 

equation. The above procedure often extends the line across 

other lines when it terminates temporarily at their crossing as 

in Pig.9 (b) where tracking along crosses many vertical 

lines. 
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4. EXPERIMENTAL RESULTS AND COMMENTS 

To test the program, experiments are made with cubes and 
wedges having relatively uniform white surfaces placed on a black 
background. The image dissector camera, used as an input device, 
dissects the scene onto a 20000 x 20000 (octal) grid. In this 
experiment, one point for every 8x8 block of grid elements is 
sampled. Thus, the scene is represented by 1024 x 1024 grid 
points. Objects occupy only a part of the scene. In the 
typical scene, the rectangular area which includes the objects of 
interest may consist of about 400 x 400 points. This area is 
divided into blocks each of which is made of 64 x 64 points and 
stored in disk memory. When a light intensity at some point is 
required, a block containing the point and adjacent blocks are 
stored in core memory. The core memory is accessed for the input 
of the light intensity until a point outside of those blocks is 
referenced. 

Video input is at first converted into a 10 bit digital 
number which is an inverse linear measure of the light intensity. 
It is again converted into 10 bit logarithmic measure. Some 
intensity level resolution is lost in the logarithmic conversion. 
In this experiment the light intensity is represented by a little 
less than ICO levels* The input data for a clear bright edge in 
the dark background is blurred due to some limitations (mostly 
defocusing). If the intensity change is a step function, there 
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is a transient area in the input data about 10 points wide. Thus 
the resolution of the picture is regarded as 10 points. The 
parameters used in line segment detection and tracking are based 
upon this resolution. Features of the picture involving 
resolution of less than 10 points are not usually found. 

Some results are shown in Fig.23. The difficulty or 
processing time of the recognition depends not only on the 
complexity of the object but also on the information extracted at 
each stage. In Fig.23 (c), for example, boundary lines SJ, KS 
and QS ar<; easily proposed as the extension of contour lines. On 
the other hand, it is not easy to find boundary lines KM or LM in 
Fig*23 (c ). That is, after DK and HL are found, circular search 
is necessary at K and L respectively. Circular search is less 
reliable in finding a line segment, and more time consuming. 
Once the boundary lines are determined, all the internal lines 
are proposed in both cases. But tracking along VW in Fig.23 (c ) 
and EN in Fig.23 (c ) terminates in the middle. Then step (10) 
stated in section 1.1 is applied. This is the most time 

consuming process (about 10 times more than the simple tracking 
process). 

Some examples of the result of a hierarchical program are 
shown in Fig.24. Hierarchical programs may look at the whole 
scene homogeneously and pick up feature points. Lines are found 
with those feature points obtained in the previous stage. It is 
very difficult to determine a priori the various thresholds for 
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detection of feature points, line fitting and connection of 

lines* In this heterarchical program, it is possible to adjust 

various thresholds with the context of the information obtained 

previously* Furthermore the algorithm itself can be modified 

case by case, (For instance, tracking algorithm is changed 

depending on whether the line is a boundary or internal*) The 

results of experiments with moderately complex scenes are mostly 

satisfactory. Because of the many checks for consistency of 

lines and vertices, the program has small probability of finding 
false lines. 

However, there are some limitations of this program at 
present. One of them is that bodies may be missed in some 
cases. A simple example is shown in Fig.25. The boundary lines 
AB and EC in Fig.25 (a) are not proposed though the other contour 
lines and internal lines are found, because the resulting regions 
are so neat' that no conceive vertices activate step (1). In 
such a case when bodies are neatly stacked, it is necessary to 
search for boundary lines which start from some points on the 
boundary line. In Fig.25 (b) body B2 is not found. To find a 
body that is included in a face of another body, it is necessary 
to search for line segments inside the region. Though these two 
kind of search (search along the boundary line and search in the 
region) are required to find all the bodies in the scenes as 
shown in Fig. 25, they are still more effective than the 
exhaustive search in the entire scene. Besides, it is simpler to 

Preceding page blank 
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interpret the scene when a line is found by those searches. This 
procedure, however, is left to future work. 

The other limitation of the present program is, as stated in 
the introduction, that it is not always applicable to concave 
objects. Fig.26 (a) shows a simple example. Line ED is found 
as an extension of line CB. If all the bodies are convex, line 
BD is interpreted as the boundary line as shown in Fig.26 (b). 
This does not hold for concave bodies. In this program, line BD 
is regarded as a boundary line, and then line DE can be found by 
circular search at D. At this stage, however DE should be 
interpreted as an internal line of the same body instead of the 
boundary line which seperates .the body into two. If DE is 
interpreted correctly, then line BD can be determined as an 
internal line. This procedure should also be implemented in the 
present program. 



CONCLUSION 
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A heterarchical program to recognize pclyhedra is presented. 

The program is based upon the strategy of recognizing objects 

step by step, at each time making use of the previous results. 

The order of the lines to be detected is 1) contour lines 

(boundary of bodies and the background), 2) boundary lines which 

are the boundary between two bodies, 3) internal lines 

(intersection of two faces of the same body. Among boundary 

lines or among internal lines, the 'most plausible lines' are 

proposed at each stage and an attempt is made to find the line. 

To find a line, the range where a line segment may exist is 

proposed and it is detected in a suitable way for the proposed 

range. If a proper line segment is found, the end of the line 

is determined by tracking along the line. When the line is 

determined, the program tries to understand the scene taking this 

line into consideration. Because lines are mostly proposed 

instead of found by exhaustive search in the scene, the program 

is relatively effective. Results of the experiment using an 

image dissector are satisfactory for scenes including a few 

blocks and wedges. Although the present program has limitations, 

some of them may be overcome by developments proposed here for 
future work. 
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LINES, LINE-SEGMENTS, REPRESENTATIONS AND CALCULATIONS: 


There are many ways of representing a line by an appropriate 
equation, A minimum of two parameters is required. Unfortunately 
these parameters tend to become singular near certain angles. 
Special purpose tests must be made to handle these cases. 

The more redundant 3 and 4 parameter representations not only 
solve this problem but simplify the operations required to 
manipulate lines and points. 

Let 0 be the inclination of the line to the x-axis andits 
perpendicular distance from the origin. ' 

Let Ow*) , (X|, y # ) and (x^,y^) be points on the line. Then 
some equations for a line are: 



y - (tan 0)x 


x 


- (cot 0)y -h 


cos 0 

/» 

sin 0 


0 




x-x 


I 




y-y, 


y-y 

at 


(x-x, )(y^-y ) - (y-v ( ) (x v -x ,) 0 


Let t jb^x^-x ,)*" ■+• (y -y cos * 


x^-x, 


sin 0 r 


y -y 


(x-x*) sin 0 - (y-y*) cos 0 * 0 
x sin 0 - y cos 0 +^ 25 . 0 



-(x^sin 0 - y 0 cos 0) 


This last formulation is nice from a number of points of view. It 
is always non-singular, easy to use and allows the least-squares 
equations to be femulated neatly. Note that we have a choice of 
polarity, by multiplying the whole equation by -1. We can choose 
a preferred direction along or across the line in this fashion. 
This allows us to represent directed lines. 

Now we get on to using this representation. First we note that the 
equation of a line at right angles through a point (x^,yj) is: 

(x-x^) cos 0 + (y-y s ) sin 0^0 
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Our line and one point (x^.y,) on it implicitly define a coordinate 
transformation: 


x* 


cos 9 sin 0 


x-x 0 

y' 


sin 0 -cos 9 


y-y B 


The inverse is: 


rx-x. 


y-y* 


cos 0 -sin 0 


x' 

sin 0 cos 0 


y’ 


The separation (perpendicular to the line) of two points: 


sin 0 - (y -y ) cos 0 


(A) 


In particular the distance of a point from the line is : 


x sin 0 -y cos e +/» 


Not surprisingly the equation shows the line, to consist of points 
with zero distance from the line. 


The separation (along the line) of two points: 


( x 2 _ -x i) cos 0 4- (y^-y,) sin 0 
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In particular suppose that the end-points of a line segment are (x^ ,y ( ) 
and (x,,y z ).Then a point lies in the band generated by projecting 
this line segment perpendicular to the line if: 

£(x-x,) cos & •+ (y-y,) sin 0 1 f (x-x^) cos 0 +(y-y t ) sin ©3<0 

This can be used to determine if a point on the line lies within 
the line segment from (x t ,y,) to . 




>o« * 


M 


bcXKol 



The sine of the angle between two lines: 

A « sin(0n.-0 x ) as sin 0, cos 0 t - cos ©, sin 

We use this in the equation for the intersection of two lines: 



(cos 0^/®* - cos )/ A 

(sin - sin ) / A 


Naturally if A« 0, the lines are parallel and we lose. 

To find out if two line-segments intersect, we use these equations 
to find the intersection of the corresponding lines, then apply 
the above "band" test twice to see if this point is inside both 
line-segments• 


Nextjto project a point perpendicularly onto the line we perform 
two opposite rotations about the origin: 


x , * x 0 cos 0*4 y 0 sin ©■ 


f x v r x , cos 6 -y f sin 0 
y^ « x x sin 0 4 y f cos 0 
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Next we get to least-squares fitting a line to a set of points: 



We minimise the sum of squares of perpendicular distances of the 
points from the line (moment of inertia): 


d 2 
dye* 6 


= z (x^ sin 0 - y^cos 0 r 


- 2 sin 0 T"* x. 

mrnm hi 

L ; 1 


) 


- cos 0 


ZZyi +/• ? l l 


For this to be 0 we must have: 


P e ~C' x o si n y 9 cos 0^ 
x # % IIx. /n 


where 


. -IVi /n 


That isjthe line passes through the centre of gravity of the points. 
Changing coordinates to a system relative to this favoured point we get: 


x! 

1 


x.-x 
1 o 


y l * y r y 0 


2 t^ x i~ x 0 ^ sin & ~ ( y i- y G ) c °s q-3 



7 -(x* sin 0 - y* cos 0 ) 

L ^ JL 


2 ^ x» « 

(sin 0) ZL x’ -2(sin 0)(cos ©■) 2Lx!y! . ( cos «) 2 T» v < 

i 1 * Ii* “u 

1 & 


* 1/2 [(^ x i + ? y r)- (f ) coa 29 - 2 l]y| 


sin 20 
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Note that we can find some of these terms as follows: 


x. « 

1 


2 2 
x . - n x 

t 1 ° 


y 


! — 


I>!y! - Z: 


1' 1 


x.y. - n x y 
1 l o y o 


For compactness let: 


a: 2 XTxIy T 
♦ 

i 


i y i 


■ZcMy* 

* 1 *1 

i I 


\ 2 ,2 A 

7 a] a 4 b 


We get: 


d0 


€ ~ b sin 2fi - a cos 29* 


For this to be zero we must have: 


tan 20 s. a/b ^ «.£. sin 20 *■ a/c 


Now 


cos 9c 


J 1 4 cos 20~^ / b 4- c ^ 

2 ~ V 2c 


COS 20 r b/c 


And 


sin 0 ~ 


sin 20 
2 cos 0 


(a/2c) / 


r f b-f C V 

V 2c 


If asO, so is sin 0. Again we can choose to multiply the whole thing by -1 
to decide the direction. 


An equivalent way of getting this last result is the following: 

■t 

^ r _ — c-b 

tan 0 s5 


-14 ^/(tan 26)1 -b4/a 2 4 b 2 

(tan 20) a 


Ob 


cos 0 * 1/J(l4 (tan 0)^ ~ 


^ ====s=s . , _. «g; a 

Ja +(b-c ) 2 ' J^cCc-b) 


_ - / b 4 c ) 
n V 2c 







^>92" 

Different least-squares fits can be described (for example one which 
minimises the sum of squares of distance parallel to the y-axis), but 
this one has the property that the fit does not depend on the coordinate 
system (invariant with rotation for example). 


Note that while trigonometric functions appear all over these results, 
none ever get evaluated. Trigonometric functions are a most useful 
intellectual crutch; there is, however, seldom a need to actually 
use them in numeric calculations. One can usually replace them 
using the well-known relationships amongst them and reduce the required 
operations to +, — , / and a/ ' only. In the above formulas 

for least-square fitting for example, the only requirement is for two 
square root evaluations. 



We ought to also check that we in fact have a minimum: 




2 2 --I 

2 b cos 20 «+■ 2a sin 20—2 a , b^ 


0 


> 



So indeed we have a minimum. We might also want to know the average error, 
and the average error if, instead, we had chosen the worst line (one at 
right angles to the best line). 



s 1/2 ( d-c ) 


e\ s 1/2 ( d+c ) 

The ratio can be used as a "form-factor". 

Note that all of this line-fitting can be modified to handle weighted 
points by simple multiplying the coordinates (x.,y.) by the weights w., 
and using 21 w. instead of n. 11 1 

i 1 
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Now suppose we are given several lines and are required to find 
a point with minimum sum of .squares of perpendicular distance. 



s 2 (-xI sin 0^ cos Q^yXCcos © i ) 2 *“ 3T P ^ COS 0_^’ 


Let A ~X(sin 0.) • 2L (cos 0.)^ 


( ZT sin 0-. cos 0.) 

i 1 1 


This will only be zero if all the lines are parallel. Solving the 
above set of equations in x and y we get: 


0 f(toS 27^sin 0^ 


Zsin 0 , cos 0 . z> 

.cos 0 . 1 / A. 

^ i i | / i i J 


j^£(sw © i cos 0^) ^® i sin 6^^ 4. X(sin 0 1 > 2 X^Vcos 9 i Jp ^ 


We also ought to check whether this gives us a minimum: 


2 * 2 2 (sin 0. ) 2 


2 

d 2 

_* e 


2 r<co S ».) 2 

( 1 


P ;> 0 


We can weight the lines by simply 
multiplying sin 0.., cos 9^, p. 
by the weights w.. 


y) 
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PROJECTION OF A RECTANGULAR CORNER: 



Given that the tri-hedral vertex is formed by three planes meeting 
at right angles, find the inclinations of the three lines 
relative to the image plane. Let these angles be . Note that 

the angle between the plane containing relative to the view vector 
is also <tand so on for the other sides of the object. 

This information is useful in defining the elevation and rotation 
of the eye relative to the coordinate system implicitly defined by the 
rectangular object. The angles can also be used to correct the fore¬ 
shortening introduced by the inclination of the lines relative to the 
image plane. 

Now consider the following spherical triangle: 
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Using a spherical trigonometry formula we get: 

0 = cos”^ = cos(^ -a) cos(r -b) + s i n (? -a) sin("^ -b) cos C 
2 2 2 2 2 

= sin a sin b + cos a cos b cos C 

cos C = -tan a tan b and by symmetry: 

cos B = -tan c tan a 

cos A = -tan b tan c 

? cos B cos C 

tan^a - - — —— -.— 

cos A 


cos a = 

Where a<0 if and only if A > If . 

So we have the angle of the line ot relative to the image plane. As 
mentioned this also gives us the inclination of the plane containing 
and relative to the view vector. The others are found by symmetry. 

To get the "unforshortened" length of the lines, that is the length in 
the image if they had been oriented at right angles to the view vector, 
we just divide by the cosine of the inclination angle: 

c* u = / cos a 

We also note that the cosines needed in the formulae can be obtained 
simply by using dot-products: 

COS A = (/&•%) /y/'ifc.fe') CHY 

So we don't need to use trig-functions at all, only +,-,*,/ and ^ ' . 


1 


cos A 


; 


1 + tan a 


cos A - cos B cos C 
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A few other random formulae in this relation: 


Since A + B + C = 2ir , cos A = cos B cos C - sin B sin C 


cos a = 


cos A 

sin B sin C 


because of that. 


sgn a = 


tan a 
1 + tan*"a 


! cos B cos C 

wmmammmtmmmmmmmmmmmammmmmmmmmtmmmmmm 

cos B cos C - cos A 


cos B cos C 
sin B sin C 


Another derivation not involving spherical triangles is as follows: 




42 


CoS \ 






Using the formulae for plane triangles: 


y 


(sin a - sin b) 2 + x 2 - 2 
2 2 2 

x = cos a + cos b - 2 cos a cos b cos C 
2 = (sin 2 a + cos 2 a) + (sin 2 b + cos 2 b) - 2 sin a sin b 


- 2 cos a cos b cos (? 


cos C = - tan a tan b 


as before 
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RELATIONS BETWEEN SIDES AND ANGLES OF ANY PLANE TRIANGLE: 



2 2 2 

a = b + c - 2bc cos A 
a = b cos C + c cos B 

RELATIONS IN ANY SPHERICAL TRIANGLE: 



sin A sin B sin C 

sin a sin b sin c 

cos a = cos b cos c + sin b sin c cos A 
cos A = - cos B cos C + sinBsinC cos a 

(a, b, c are the length of the arc on a unit sphere* alternatively 
one can think of them as the angle (in radians) subtended by the 
arc at the centre of the sphere) 
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PROXIMITY FINDING: 


In the later phases of line-firiding programs it is often necessary 

« 

to repeatedly locate lines that are close together, lines that pass 
close to a given vertex and so on. To do this efficiently we require 
a fast access method to locate likely candidates for more sensitive 

tests. First consider the problem of deciding if two points are close 
together. 


To tell if x and y are close together we can quantise both (by dividing 
by the quantisation interval size and truncating). If the two numbers 
[x/d] and [y/d] are the same we win, but it may be that x and y just 
straddle a boundary defined by our truncation algorithm. We need 
a second, interlaced set of boundaries and calculate [x/d+.5], [y/d+.5]. 
Now if either pair of numbers matches we know that |x-y|<d. Conversely if 
|x-yKd/2 we are guaranteed that at least one pair will match. 


This method only comes into its own if we have large sets of points. We 
then simply find the two integer-codes for each one and add it to the 
appropriate buckets. To find which points are near a given point we 
determine its two integer-codes and collect the union of the two 
corresponding buckets. 

__ » l 

. . """ " " » . — - » , ■■■ ■ - - 
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This method can now be extended to n dimensions. We need at least n+1 sets 
of buckets if we use n-tetrahedrons. It may be more convenient to use 2 n 
sets of buckets if the unit cell is an n—cube.(d/n versus d/2 min sep) 


Line-segments and curves can be handled by entering each point on them 
into the system. In practice one will only enter a set of points separated 
by the minimum distance guaranteed by the geometry used. Retrieval works 
in a symmetric way. 
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LEAST SQUARES SOLUTION OF AN OVERDETERMINED SET OF EQUATIONS: 
Let us write the equations as follows: 


A x = y + e 

Where A is a given m by n matrix ( ,m>n) 5 x is the unknown n-vector, 
y is a given m-vector and e is an m-vector of errors which we are 
trying to minimise. 

e T e = [Ax - y) T (A x - y) 

= (x T A T - y) (Ax- y) 

T T T T T T 

-xAAx-yAx-xAy+y y 

— e T e = x T A T A + (A T A x) T - y T A - (A T y) T + 0 
dx 

So: x T A T A = y T A for this to be 0 

A T A x = A T y 
x = (A T A) -1 A T y 

T T 

— e‘e = 2 A ( A 

dx'* 

The diagonal elements of this will clearly be positive so we do have 
a minimum. 


• * 
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LEAST SQUARES CURVE FITTING: 


Suppose we have a function g(x) defined at the n points x 1 , x 2 etc., 
and that we are trying to fit a function f(x) which depends on the 
m parameters a-j, a 2 etc. so as to minimise the sum of squares of 
errors at the points x ] , x 2 etc. Let e. be the error at point x.. 

e. = f( Xi ) - g(x.) 

Let e be the n-vector of errors, f the n-vector of fitted values, the 
n-vector of defined values. Then we are trying to minimise: 

e T e = (f - £) T (f - g) 


by varying the parameter m-vector a. The derivative of e T e. w.r.t. 
to this vector must be zero (and the second derivative positive). 


df 

(f - £ ) — = 0 

da_ 


df 

where —■ is an n by m matrix 

da_ 


df 

df 

So: i — 

= f ...» 

da 

da 


or written out in full: 




M(*h) 

%(*u> 


MW Mfo) 


4M 


^ | » 

M, 




fa) 






cM6**) AXfxi.') • 

MM 

eU.* 




S 


A* t, 

ft 


1 « 

1 

MM , 

* 

MM 



* • • 

a M 


fa) 

cLo.*a 

Mi» 







To be able to solve these equations we choose a particularly simple 
form for f(x) namely 21 a J f J fx 1 ) = f(x,) , that is a linear one. 

In this case the terms in the matrix df/da, namely dfCx^/da. become: 
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df. 

— 1 = f j (x i ) 

da. ^ 

J 


So the matrices become quite simple and let us denote them by f" 1 " 


df. 

J 


We also note that because of the simple dependence of f(x) on the 
parameters a. we get: 


f = F a 


And we can rewrite the main equation as: 

F T £ = F t F a 
T 

Since F F is square we can attempt to invert it and get: 

a = (F^F)- 1 f t a 

So inverting the normal matrix allows us to solve for the parameters a. 
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EXAMPLE OF FITTING A STRAIGHT LINE: 


f(x) = a-, + a 2 


x 


and let y i = g(x i ) 


Here 


a= (a-j, a 2 ) £ = (yj,y 2 , y n ) -f-, (x)=l f 2 


II 

1— 
Ll. 

11 ... 1 1 

f^"f = 

n £x. 


X 1 x 2 •• x nl 


I x i £ x i 


O p 

Let A = nTxf - (Tx.) , then 


(F T F) _1 


\ 

A 


rx? -Zx, 

,-T 


-Zx i n 

F £ = 

iVi 


a = (FT) 


1 F T n _ * 

F £ = —- 

Ixf -Zx n . 

» 


Zy i 

A 

n 


^ x i y i 


a 1 = ( lx? Xy i - Xx i X x^.) /A 
a 2 = ( -ZxiZ. y i + n 2 x 1 -y i ) / A 

Note that this is an unsymmetrical method different from the one 
demonstrated elsewhere in this memo and not suited for fitting 
lines in a line-drawing, for example. 


(x)=x 
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APPLICATION TO FITTING A POLYNOMIAL: 





and let y i = g(x..) 


a = (a Q# a 1 ... a m _ 1 ) £ = (y 0 , y, > ... y n ) 

fj(x) = x j 



1 1 
X 1 x 2 


1 


• • • 


n 


x 


m-1 

1 


• « • X 


m-1 

n 


f t f 




• * 



* 



The "normal" matrix 



At this stage we obtain the parameters a= (F T F) _1 F T £ 




m 
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FITTING EXPONENTIALS 


f(x.) = 2- a i( s i) 

1 j*o J J 


f ,(x) = (Sj) 


a - (yij.y-i. y„-i) 


n-1 


n-1 


*o *1 

Vl s m-l 


n-1 

'm-1 




(F T F),, = 21(3,3,)* 

‘j y.*t) 1 J 


Now suppose we have regular intervals x,=kT 


Let sj = sT 


. _ ST 


(f t f).. = 21 < s i s 1 ) ,<T = Z- (s!sn 

J K*o J 


and s'. = s. 
J J 


(1 - (s!sj) n )/(l-s!sl) 


Next take the special case: 


s , = e (2*i/n)a -(2wj/n) 

9 


(F F)..=0 for i+j70 or n 

' vJ 


(F F)..=n for i+j= 0 or n 

' vJ 


(F T F)= 


1 0 0 
0 0 0 
0 0 0 


0 0 
0 1 
1 0 


(F T F) _1 


1 0 0 0 0 
0 0 0 0 1 
0 0 0 1 0 


0 1 0 


0 0 


0 10 0 0 


*. * (r T ?i 


r w- 0k y k 

Z*w' lk y k 
XW (n - 1)k y k 


r ,0k 

w y k 

1 ITw ,k y k 
2> (n ' 1)k y k 


f n _ a )k — a k 

Where we used w v = w Note: a is the discrete F.T. of cj_. 
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APPLICATION TO FITTING NON-HARMONICALLY RELATED SINES AND COSINES: 


Assume regular sample intervals: x.. = iT . Let s. = ^ n ^t, where 4^ 
are the frequencies of the various components. The s. should be non-zero, 
positive and unique. 


f(x.) 


a + 
o 



a. cos(s . i) 

J vJ 


r* 

+ 21 b. sin(s. i) 
J J 


f r s 


1 1 


1 


1 


1 COS(s-|) COs(2S-|) 

1 cos(s 2 ) cos(2s 2 ) 


* • * 


k » I 


cos((n-l )s^) 
cos((n-l)s 2 ) 


» * 


• * * 


1 

0 

0 


cos(s m ) cos(2s m ) 
sin(s^) sin(2s,) 
sin(s 2 ) sin(2s 2 ) 


» * * 


* * • 


» 11 


cos((n-l)s m ) 
sin((n-l )s,) 
sin((n-1)s 2 ) 


♦ * I • 


» * * 


0 sin(s_) sin(2s m ) 
m m 


* * * 


sin((n-l)s ) 


Now note that 


2 cos A cos B = cos(A+B) + cos(A-B) 
2 sin A cos B = sin(A+B) + sih(A-B) 
2 sin A sin B = cos(A-B) - cos(A+B) 


and let s =0 
o 


(F T F) . . = X cos(s,k)cos(s .k) = (1/2)2! cos((s.+s .)k) + X cos((s.-s.)k) 

1 K-o 1 J 1J J 1 

for 0 4i ^m and O^j^m 


(FTf:) i J j+ m = <£cos(s.k)s1n(Sjk) = 0/2) Xsin((s.+s.)k) +Z sin(( Sj -s .)k) 

for 0 4i4 m and 1 ^ j^m 




( pTp )i+ m j-Hn = ^ sin ( s i k ) sin ( s j k ) = (1/2) XIcos((s .-s. )k) -Zcos((s.+s,)k) 

K* ( vJ * O 

for 1£ m and i<o4 m 

The other terns can be found using the symmetry of (F T F). 
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Now 21 e jwk = (1 - e Jwn )/(l - e jw ) 

Vlco 

= ( e J' wn / 2 / e J w/2 ) ( e J ' wn/2 _ e -J‘ wn / 2 )/( e j w / 2 _ e “jw/2j 

= e^ w ^ n_1 ^ 2 sin(nw/2) / sin(w/2) 

Now since cos (A) = Re ( e J ' A ) and sin(A) = Ig ( e J ' A ): 

V\~ \ 

ZI cos(wk) = cos(w(n-l )/2) sin(nw/2) / sin(w/2) 

= (1/2) ( sin(w/2) + sin((2n-l)w/2))/ sin(w/2) 

= (1/2) (1 + sin((2n-l)w/2) / sin(w/2) ) 
unless w =0 in which case the sum is n. 

2Hsin(wk) = sin(w(n-l )/2) sin(nw/2) / sin(w/2) 

ILco 

= (1/2) ( cos(w/2) - cos((2n-l)w/2))/ sin(w/2) 
unless w =0 in which case the sum is 0. 


(FF) i,d 


= —(2 + 
4 


sin((2n-l)(s.+s.)/2) sin((2n-l)(s.-s .)/2) 

__ * J i I . I 


4 sin((s i -+s j .)/2) sin((s^-s -)/2) 

for and 0-4j^m and i/j. If i=j, the third term is 2n-l. 

for i = j = 0, the second and third term become 2n-l. 


(F T F). = — ( 
»J+m 4 V 


1 cos((s.+s.)/2) - cos((2n-l)(s.+s.)/2) 
= —- L A - 1 J 

4 sin((s i +s j .)/2) 


cos((s.-s.)/2) - cos((2n-l)(s.-s.)/2) 

4 _!__ _ I ^ 

s1n('(Sj-s i )/2) 

for O^i^m and l4j^ m and i /j. If i=j, the second term is 0 

. T . 1 sin((2n-l)(s.-s.)/2) sin((2n-l)(s,+s.)/2) 

(F F)., , = (+ - J 1 -- - J 1 \ 

/ sin((s J --s j )/2) sin((s i +Sj)/2) ’ 

for l^i^m and l^j^m and i/j . If i=j the first term is 2n-l. 
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The first element in this array is n, the other diagonals are near n/2, 
while most of the other terms are small, of the order of 1. The only 
large elements will be the result of two narrowly separated frequencies. 

This makes for good numerical stability when inverting (F T F) by the 
simplest methods. 

When the frequencies are harmonically related , we have s.=(2TT i/n). 
Then all off-diagonal terms will be 0, and those on the diagonal will 
be exactly n/2 except the first which will be n. The inverse of (F T F) 
then is also diagonal with the first element 1/n and the rest 2/n. We 
are back to discrete fourier transforms in this case. 

Note that if we use: 

sin(A+B) = sin A cos B + cos A sin B 

sin(A-B) = sin A cos B - cos A sin B 

cos(A+B) = cos A cos B - sin A sin B 

cos(A-B) = cos A cos B + sin A sin B 

we can obtain all the entries in the array using only a few operations 
on sin(s i /2), cos(s.j/2) and sin((2n-l )s^/2), cos( (2n-l )s ./2) . 





Page 264 


SOLVING SETS OF SIMULTANEOUS LINEAR DIFFERENCE EQUATIONS: 


( x l^n+l a 11 ^Vn + a I2 ^ x 2^n *“ + a lm^ x m^n 

^2^+1 = a 2I ^n + a 22 ^ x 2^n — + a 0m (x 1 

2m m n 

i *» 


(Vn+l = a 


ml + a m2 ^ x 2*n 


(x 0 L + 

2 n mm m n 


)c + 1 = A x where A is the given coefficient set 

Assume a solution of the form r 11 for each x.: 

where a is a m-vector of parameters 

(A-Ir)a^ = 0 since r 11 f 0 

A non-zero solution for £ requires that det(A-Ir) = 0. The possible 

a_'s are eigenvectors, the possible r's are eigenvalues of the matrix A. 

The determinant is a polynomial of degree m in r and will usually have 

m solutions, possibly complex. We get the usual problems if two roots 

coincide and have to introduce additional solutions of the form nr 11 , 

2 n 

nr and so on. Having found r, we can solve for a using some 
normalising conditions (since any multiple will also be a solution). 
Then using the linearity of the set of equations we can add up the 
solutions into a more general one: 

x -Z a. (r.) n where r. are the various solutions 

HI jsi -J J J 

Often we are only interested in stability 



and just check the roots 
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EXAMPLE FOR A TWO VARIABLE SYSTEM: 

^ X l^n+1 = a ll ( x l) n + a 12^ x 2^n 
^ x 2^n+l = a 21 + a 22^ x 2^n 


det 


(a-j i - r) a 


12 


21 


( a 22 " r ) 


~ ( a 11-i") (a^-r) “ a-]/5a 01 = 0 


22 


12 21 


( a H a 22 " a i? a ?i) “ ( a n + a 99 )r + r 2 = 0 


12 21 


11 22 


*" ~ ( ( a n +a/ jo) i \j ( a n +a 22^ "^^ a ll a 22” a 12 a 21 ^ ^ 


11 22' - s ( vu ll 1 “22 

r = (1/2) ( (a^+a,,) + *J ( a n - 


11 “22 

Stability: When is 


11 “22 


2 \ 

a oo) + 4 a 10 a 01 ) 


12 21 


is \ (a + J a 2 - 4b ')| <X ? 


Case 1: 


4b> a‘ 


Complex roots 


l a l < 2 for stability 


Case 2: 


4b < a 1 


Real roots a> 0. a+J a -4b { 2 




a< b+1 

Case 3: 4b^ a 2 Real roots a^O. 

-a+J a 2 -4b < 2 


-a <b+l 

Case 2 &3 4b {a 2 Real roots 

|a\^b+l for stability 

Substituting: 


Case 1: 4 ^ a ll a 22" a 12 a 21^ ^ a ll +a 22^ 2 


-4 a-|2 a 2l > ^ a n" a 22^ 2 

l a H+ a 2 2 l< 2 

Case 2 & 3: Opposite of above relations 

l a ll +a 22l <1 + ( a n a 22" 
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MULTI-DIMENSIONAL NEWTON-RAPHSON ZERO-FINDING: 


Suppose we have nfunctions F.. each of n parameters a. . We are trying 

to find values for the a.'s such that the F.'s are all zero. 

J * 

Assume we have the value a at step n for the parameter vector. 

This gives us the value for the function vector F = F(a ). 

—n —n 

Now consider a small change da in a_. To a first approximation we get: 



Where F'(a ) is the matrix of derivatives: 


dF, 

dF i 

dF, 

da-. 

■* i 

d&2 

da 

n 

dF 2 

dF 2 

* • » 

fn 

da. 

da 2 

da 2 

» % 

1 

»«* 

Z” 

dF n 

« % , 



da, da 9 da„ 

r 2 n 


+ da) = 0 we need -F(a ) = F'(a ) da 

~fl —n — 

d — * - w' f „ 

Vi - ^ - (W 1 F „ 


For F(a 
—n 


So 


So we iterate to a solution, requiring one matrix inversion per step. 
There are better methods, but few simpler. For bad hi 11-climbing type 
problems one can use the method of conjugate directions and various 

variations such as the so-called mixed method which is also fairly 
rapid. 
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SIMPLE INTERPOLATION OF A FUNCTION FROM A STORED GRID: 


Rectangular grid: Suppose the origin is x Q , y Q and the spacing d. 

To find an interpolated value at a point x, y calculate as follows: 

x' = (x - x Q )/d y' = (y - y 0 )/d 
* = Ex'] j = [y*] 

A i = x 1 - i A j = y' - j 


f(x,y) =* f isj (l-Ai- Aj- Ai Aj)+f i+1>j Ai(i-Aj) 


+f i,j+l^" Ai ^ J ' + f i+l ,j+l 


Triangular grid: Again origin at x Q ,y 

IT » 

x * (—(x-X„) —-(y-y„))/d y'-ty-y. 




Al= x '-i A j = y' - j 
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WHAT ELLIPSE IS IT: 

Given an ellipse in the form: 

A x t B xy + C y 2 +Dx+Ey+F=0 
Determine its center, angular orientation and major and minor axes. 

x Q = (BE - 2CD)/(B 2 - 4AC) y Q = (2EA - DB)/(B 2 - 4AC) 

x , y Q are the center because we can expand as follows: 

A(x-x Q ) 2 +B(x-x Q )(y-y Q )+C(y-y Q ) 2 +F 1 =0 

A x 2 + B xy + C y 2 + (-2Ax Q -By o )x + (-2Cy 0 -Bx Q )y 

+ (F'+AxQ+Bx o y o +Cy 2 ) = 0 

So we have: 

2Ax„ + By„ = -D 
o o 

Bx„ +2Cy„ = -E 

o o 

Solving this set of equations we get the above expression for x , y . 

o o 

We also now have a useful new quantity: 

F' = F -(Ax 2 + Bx v + Cy 2 ) 

v 0 O' 7 0 J 0 ' 

The orientation of the ellipse is found as follows: 
tan 2 9 = B/(A-C) 

And the major and minor axes can be found as well: 
a*"= -2F7((A+C)-J B 2 +(A-C) 2 ’) 
b t = -2F’/((A+C)+ yB 2 +(A-C ) 2 ") 
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These last results follow from expansion after change of coordinates! 


x cos S + y sin 0 , -x sin 0 + y cos 0 „ 

(- )2 + (- )2 = ! 

a b 


sin 0 cos 0 


) x + 2 sin 0 cos 0(-« - -«) xy + ( 

a^ b^ 


s i n 0 cos 9 


“T" + —jr)' y = 1 

a b c 


Identifying the appropriate terms with A, B, C and F 1 we get: 


B/(-F') 


= sin 29 ( 




(A— C)/(-F 1 ) 


= cos 29 (• 


7?) 

b^ 


Since 2 sin 0 cos 9 = sin 20 and (cos 0) 2 - (sin 0) 2 = cos 20 


(A+C)/(-F') 


1 1 
= ^ + "2^ 


Its also clear that: 


."9 " "9 

a^ b^ 


B 2 + (A-C) 2 X / 


(Assuming a > b) 


The rest follows from these simple equations. 
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APPROXIMATION TO ni 

Stirling's formula: nl * \2t n n (n/e) n 

__ (1/12n) 

Better approximation: ni v A 2nn* (n/e) e 

The fractional error of the latter is much smaller. For n=10 for 
example it is .27E-5 versus .8E-2 and for n=50 it is .221-1 versus .IE-3. 
This is useful in calculating large binomial coefficients. 

OBTAINING A NORMALLY DISTRIBUTED RANDOM VARIABLE FROM ONE UNIFORMLY 
DISTRIBUTED: 

Suppose x. is the output of our random (pseudo ...) number generator. 

»+N 

1. (X X. - 6)/6 

;«* 1 

2. J-2 log x? sin(2Tfx i+1 ^ 

3. Let f(x) be the distribution we are aiming for. Now integrate it: 



A random variable distributed as desired will be (F ')(x.) 

MULTIPLICATIVE RANDOM GENERATORS: 

x. +1 = A k x.. (mod p) pa large prime 

A a primitive root of p, k not a factor of p-1. 

Example: p = 2 33 -31 A=5 k = 5 

p=2 31 -l A =7 k = 5 

Additive congruential generators are better, though. 
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FAST IN-POSITION MATRIX INVERSE: 


Do 1=0(1) n-1 
com 4 a(i,i) 
a (i, i) <1. 

Do j = 0 ( 1 ) n-1 
a(i,j) 4 a(i,j) / com 
End 

Do k = 0 ( 1 ) n-1 and i f k 
com 4 a(k,i) 
a(k,i) <• 0. 

Do j = 0 ( 1 ) n-1 

a(k,j) 4 a(k,j) - a(i,j) * com 
End 



End 


Note that rows and columns are never shuffled and that there will be 
matrices which while not singular will cause this procedure to fail. 

The matrix is n by n and stored in the array a(i,j) where i and j 
range from 0 to n-1. 

GENERATING A BIT-REVERSING TABLE: 

b(0) 4 0 
m <■ 1 

Do i = 0 ( 1 ) 1n-1 

Do j = 0 ( 1 ) m-1 

b(j) 4 b(j)*2 

b(j+m) 4 b(j)+l 

End 

m 4 m*2 
End 
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SOME FOURIER TRANSFORM METHODS FOR IMAGES: 

Fourier transforms for images are two-dimensional and two-sided. In this 
they differ from time-series type transforms which are one-dimensional 

and often pertain to one-sided functions (impulse responses must be 0 for 
negative values of time). 


The general formula for n-dimensions is: (Note: f and g are complex) 


1 



e _i 





g(iO e +1 


u .x 



Where x_ is the n-dimensional source-space vector, u_ is the n-dimensional 
transform-space vector and g is the transform of f. For two dimensions: 




oG> 

A A 



g(u,v) = 

1 

“ / / f(x,y) 

e -i(ux+vy) 

dx dy 


2 TT 

-hO -oG 


f(x,y) = 

1 

• rC g(u,v) 

e +i(ux+vy) 

du dv 


2 t r 

J J 

•0O -oO 




Many functions of interest are rotationally symmetric and can be dealt 
with by use of the one-dimensional integrals obtained after introducing 
the polar coordinates (r, d) for(x, y)and for(u, v). 

9(f) = J 9 r J (>( r f ) dr 

f( r) = /%</•) f y-r ) if 

Where J Q is the zeroth order Bessel function. 

Note that f and g are now real-valued. 
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This follows from: 


1 

2 IT 

1 

2 ir 

i 

2ir 


5° -itr 


y y f(r) e _i cos / cos e + r^sin j sin 6) 


d0 dr 


L 


oO 


f(r) 




COS 


(/ - ») 


d8 dr 



f(r) 2-rfr J Q (rf ) dr 


We can apply these results to a few useful examples: 

The pillbox: f(r) = 1 for r 4 R, 0 otherwise. This is the point-spread 

function produced by defocusing. 

ft .ft/* 


g (/>) = / r J 0 (iy>) dr = J- J Q (* )dx = ItL 


J-, (R/>) 


. . 2 ^ 1 ^ A ) a °° 

9 r )= R (R/>) since 7. *v x) - xj i( x) 

So the function J-j(x)/x plays the role here that Sin x/x plays for 
one-dimensional systems. 


The gaussian: 


- '/*V 

f(r) = e 


9<f) 


* f* 2 i 

e " x ( r } r J Q (rx> ) dr = <r 2 e" 


Since , e 



-* x2 J n (bx) x" +1 dx = -1^ e 


(2a) 


The gaussian has some interesting properties. First it is the 
only rotationally symmetric function that can be factored into 
a product of a function of x and a function of y. Secondly it is 
the only function that "transforms into itself". 
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The gaussian is also a good first approximation to point-spread 
functions in some devices(at least for small r). 


A scatter function: f(r) = e ~" r / r 

Analysis of total reflections in the face-plate of an imaging 
device leads to an equation of this form (at least for large r). 



Note on scaling: For the gaussian we have the following relationship: 


r H f H = 1.386 (r H = 1.1770- 5 /» H = 1.177/CT) 

where r,, is the half-intensity radius in the source space, 
and is the half-intensity radius in the transform space. 
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HOW COHERENT MONOCHROMATIC LIGHT AND A LENS DO FOURIER TRANSFORMS: 



Let f be the focal length of the lens and X the wavelength of light. 

Plane monochromatic coherent light enters from the left and passes 

through the transnarency, being then focused by the lens on the image plane. 

We assume that x and u are relatively small compared to f, so that 

9 will be small. We then have for the distance that the ray has to travel 
from the point x on the source plane to the point u on the image plane: 

f/cos 0 + f cos 9 + x sin 0 = f(2 + 0 4 /4 + 0 6 /l20 ... )+x(0 - 9 3 /6 ..) 

For small 0 this is approximately: 2f + x 0 

The phase-shift in radians is then: (2f + x 0) * 2-n/\ 

We can ignore the constant part of this and considering that light 
will arrive at the point u from all over the transparancy we get: 

/ + o*> x u 

2 tt i —L 

f(x) e X * dx 

-oo 

Where f(x) is the amount of light passed through at the point x. 

Now extend this to two dimensions and we finally have: 

f T 2tr i (MlpL) 

g(u,v)-y yf(x,y)e dxdy 

—06 — ofe 

Note that g(u,v) is complex. We can get an idea of scaling from this 
equation. 
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SOME HEURISTICS FOR TELLING WHAT HAPPENS WHEN YOU TRANSFORM A FUNCTION 


Source domain 


Transform domain 


Periodic 

Symmetric (about 0) 

Non-zero for finite distance 
Compact 

Sharp transitions 
Sample of f(t) 

Sum of f(t) and g(t) 
Convolution of f(t) and g(t) 
Time shift of f(t) by 
Integral of f(t) 

Differential of f(t) 


Discrete (non-zero only for some f) 
Real 

Non-zero out to infinity 
Spread-out 

Lots of high frequency components 
Periodic copies of F(w) 

Sum of F(w) and G(w) 

Product of F(w) and G(w) 

Multiply F(w) by e^ w 
Divide by jw 
Multiply by jw 


These rules apply going either way in the transformation and may be 
used simultaneously. Discrete fourier transforms, for example, are both 
periodic and discrete in both domains. 
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MEASURING MODULATION TRANSFER FUNCTION USING SQUARE WAVES: 

It is very hard to produce images in which the intensity varies 
sinusoidally. Yet such images are required in the traditional deter¬ 
mination of frequency response or modulation transfer function. An 
alternative is the use of simple-to-produce square wave intensity 

modulated images. Then however we have to recover the transfer function 
from the measured results. 

Let t be one of the spacial dimensions and w = (2-rr)/T, where T is the 
repetition interval. The input can be analysed into: 

oo mi 

f(t) =1/2 + 21 b(n) cos n*ot where b(n) = (2/rr n^-l) 2 " 

for n odd, 0 otherwise 

Let the transfer function be a (to). Then the output will be: 

g(t) = (l/2)a Q + 21. b(n) a(n u ) cos nwt 

We can easily normalise to let a Q = 1. Let c(w) = Z. b(n)a(n w ). 

The problem is to recover a( uj ) from c( w ). In the case of square-waves: 

c(w) = (2/-ir)( a(w )-ia(3w)+Ia(5w )-ia(7w )+^a(9w )-^a(ll w) ...) 

c(3w)= (2/-r)( a(3ui)-^ a (9w)+^a(15w) ...) 

c(5w)= (2/-rr)( a(5w)-| a (15w) ...) 

c(7*)= (2/tr)( a(7u/) ...) 

c(9w)= (2/rr)( a(?ui) 

Now add appropriate high-order terms to c(uj) to cancel out high-order 
terms of a(u>) and get: 

a(u») * (*rr/2) ( c(w)+^c(3u»)-^c(5w )+^c(7u»)+—i-c(ll u> )-«l c (13 u») .. } 

3 5 7 |l | ^ * 1 
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CONVOLUTIONS OF PILL-BOXES: 
With a line: 


«r 



Clearly the convolution is 2 J R Z - r 2 ' = 2 R Jl - (jr) 2 ' for jr)<R 
This then gives us the intensity profile of a defocused line. 
Convolution of a pi 11-box with a step: 

We simply integrate the above: 



2 z 
k - dx 


? 2 - ♦ R 2 sin- 1 (£)] 


r 

-R 


1 


1 


* TR (7 


. ^,{1 /TTTp) 


So we have the intensity profile of a defocused edge. 


This can be rewritten in a slightly different form using: 


. -1 -1 
sin x + cos x = 


We can also use this to find the convolution of two pillboxes as 


| r| 

2 FC - —) 


2 
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A LINEAR THEORY OF FEATURE POINT MARKING 

A first step in many line-finding programs is a process for 
determining which points in the image are likely to be on an edge. 
This is usually done by locating areas of rapid intensity variations. 
Various ad hoc linear and non-linear techniques of varying support 
in the image are brought into play. It would be useful to have an 
anchor point on this spectrum of possible procedures. Since a lot is 
known about linear methods we might ask what linear method applied 
to a somewhat idealised image would do the job. 


Given a function f(x,y) which is constant within polygonal areas 
in the image,we are looking for a convolution function h(x,y) which 
when applied to f(x,v) will be zero everywhere except on the edges. 

00 <*> 

g(x,y) = J j f(x-x'.y-y') h(x',y') dx' dy' 

-«0 -CO 

To attempt to answer this question we might start by asking what values 
we expect g(x,y) to take on the edges. Linearity considerations dictate 
that it somehow be proportional to the intensity step. In addition it 
must reflect the orientation of the step, to insure that superposition 
will work. A combination of a negative and a positive pulse will do the 
trick, provided the area under each is equal to the intensity step. 

Since the image is actually two-dimensional we will have two pulse walls 
running along each edge. 



$Cx,y) 
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Note,by the wa^ that the regions of uniform intensity don't have to be 
polygonal. Now it is pretty hard to guess what form h(x,y) will take. 

A way to get a handle on this is to ask the inverse question: what 
h'(x,y) when convolved with g(x,y) will produce f(x,y) ? 

f(x,y) = J j g(x-x‘,y-y 1 )h 1 (x 1 ,y 1 ) dx 1 dy' 

Well, it helps to look at some simple cases first. In particular if we 
only have one contour (one closed curve made of the double pulsed wall) 
we expect to get 0 if the convolution is about a point outside this 
contour^and the intensity step if the point is inside the contour. 



A and C illustrate the above statements, while B and D are special cases 
useful for deriving equations. From D in particular we find that 
2iTr (d/dr) h'(r) = 1. This is assuming that h must be rotationally 
symmetric which is clear from the other examples. We also noted that 
convolving with the double pulse wall is just like taking the derivative. 

h'(r) = (1/2-TT ) J (1/r) dr = - (l/2ir) log r 
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Since this function also does the right thing for example B we have the 
desired result. Now we need to find h(x,y) from this. We do this by 
finding the fourier transform of h'(x,y) and noting that it must be the 
algebraic inverse of the transform of h(x,y). Since the functions are 
rotationally symmetric we get: 




H 


'(/>) =-0/2*r ) f 0 


Integrating by parts and using J 


J (x) we obtain: 


log r r J Q (r^ ) dr 

xJ (x) dx = xJ-j(x) as well as^/j^x) dx = 


H '(f>) = (l/2n^ 2 ) 


To obtain the transform of h(x,y) we just invert this: 


) = 1/H' (f ) = 2 Vp 2 

When we try to inverse transform this we get into convergence difficulties 

and soon discover that we have to expand our universe to that of generalised 

functions if we expect to win, even if we use convergence factors. It then 

also becomes reasonable to guess at the answer. Consider the sequence of 

"functions" obtained by repeatedly differentiating a unit step. The first 

is a pulse at the origin, the second two pulses of opposite sign. This 

function corresponds to (d/dx) in the following sense: if we convolve 

it with a function f(x) we obtain the derivative f'(x). Similarly, the next 

member of this sequence consists of a negative, a double height positive 

2 2 

and another negative pulse and corresponds to (d /dx ) and so on. 

When we try to transform these "functions" we obtain the following: 

T(d/dx) = iu, T(d 2 /dx 2 ) = -u 2 , T(d/dy) = iv , T(d 2 /dy 2 ) = -v? And: 

T( d 2 /dx 2 + d 2 /dy 2 ) = -u 2 -v 2 =y> 2 ! 

So our h(x,y) is some multiple of the laplacian ( d 2 /dx 2 + d 2 /dy 2 ). 
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One can amuse oneself by showing that the convolution of h(x,y) and 
h'(x,y) is in fact zero everywhere except at the origin as it ought 
to be: 

h(x,y)«h'(x,y) = (d 2 /dx 2 +d 2 /dy 2 )(-1/2-rr) log r 
d/dx log r = d/dx (1/2) log(x 2 +y 2 ) = x/(x 2 +y 2 ) 

d 2 /dx 2 log r =-(x 2 -y 2 )/(x 2 +y 2 ) 

d 2 /dy 2 log r = (x 2 -y 2 )/(x 2 +y 2 ) by symmetry 

(d 2 /dx 2 +d 2 /dy 2 ) log r =0 except for x=y=0 

So log r is the function which has the surprising property of having 
a curvature at each point which is exactly opposite to the curvature 
at right angles. Next we might be interested in a discreet approximation 
to the laplacian, particularly a rotationally symmetric one( Jb) \ 





Now since we have all this nice linear theory ala Wiener available we 
might as well mention that if the image is corrupted by gaussian spatially 
independent noise we can apply his results to produce a least squares 
approximation to g(x,y). We then find our convolution functions more spread 
out than the laplacian and in fact they will contain a central peak surrounded 
by a larger negative depression (o. The only problem is that only 
some part of the noise in the image satisfies the criterion, a great deal 
of it not being spatially independent and what's worse, there is no reason 
to suppose that a least squares approximation to our pulse walls would 
be at all useful. Anyway, here it is, our anchor point for the spectrum 
of feature point (or inhomogeneous) finders. 
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FAST FOURIER TRANSFORM 


Once we have bit-reversed the complex array x containing the function 
to be transformed we proceed as follows, assuming In = log^ n. 

n 4 21 1 n 

i tn 4 n/2 
igr 4 n/2 
iga 4 2 
is 4 1 

Do i = 1 ( 1 ) In 

Do ist = 0 ( iga) n-1 

Do k = ist ( 1 ) ist+is-1 , iwb = 0 ( igr ) 

a <• x(k+is)*w(iwb)+x(k) 
b 4 x(k+is)*w(iwb+itn)+x(k) 

x(k) 4 a 
x(k+is) 4 b 

End 

End 

igr 4 igr/2 
is 4 iga 
iga 4 iga*2 

End 

2trai 

Where w(a) = e n 

Note that the arrays x and w are complex valued and dimension n. 
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CONTRAST IN A RECTANGULAR CORNER: 

One of the problems in generating line-drawings from comolex scenes 
is that in addition to the contrast-reduction due to scatter in the 
imaging device there is also a great reduction in contrast due to 
mutual illumination. To get a handle on this problem, consider the 
simple case of two semi-infinite planes meeting at right-angles. The 
light is incident at an angle w.r.t. one of the planes. The surface 
is such that/> of the incident light is reflected. Clearly for any 
point on one of the half-planes one half is reflected into empty space, 
the rest onto the other surface. Light incident at any point is a sum 
of the light from the source and that reflected from the other plane. 

If both planes are semi-infinite the intensity on each one will be 
uniform since a point receives an amount of light from the other plane 
that does not depend on the position of the point. 



l Z = <r ^ *1 + a sin# * 


I] = (cos* + (p /2) sin*0 a / (1 - (p /2) 2 ) 
I 2 = (sin* + (p/2) coso() a / (1 - (p /2) 2 ) 


Contrast = 

h - h 


((^/l-l > GQ stf -(/54-l)sintf ) 

JJL 

coso< -sin* 


h + h 


(( f/l+V cos* + (*4.+i)si n <) 

2-f 

coso^+sinrf 
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Contrast = 


V^2 


2 - 


— I tan(tf -f/4) 

2+f 1 


In the absence of reflection this will be |tan(*4 -1T/4)|, so the 
contrast is reduced by a factor 


(2-p)/(2+f> ) 

This factor ranges from 1/3 to 1 as P ranges from 1 to 0. So for 
objects that reflect most of the incident light, such as our white 
cubes this effect is worst, reducing the contrast by a factor 3. 

If we consider finite half-planes things get more hairy and the 
intensity on a given plane is no longer independent of position, 
falling off as one goes outward from the corner. In the corner 
itself however the situation is unchanged in the limit. So as far 
as the contrast across the edge in the image is concerned we can 
still use the above formula. Note that with finite half-planes a 
rigorous analysis would require knowledge of the distribution of 
reflected light with angle which was not needed in the above. 


If we consider other angles we find that the problem increases as the 
angle gets smaller. 


Suppose the angle between the two planes is ■tf/k instead ofir/2. Then 
instead of if/2) we have (1-1/k)^ . The reduction in contrast then is: 

1 - (1 - 1/k)/» 

1 + (1 - l/k)f 

And whenP = 1, the worst case,we have a reduction of l/(2k-l). 

It is clear that gray blocks are very much better in this respect than 
white ones. For example ifP= .5 instead of 1.0, the reduction is only 
3/5 instead of 1/3 for the contrast. 
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SCATTER IN OUR IMAGE DISSECTOR (TVC): 

A considerable reduction in contrast in our image dissector is 
caused by scatter of the incident light. This scatter goes undetected 
when one concerns oneself with the point-spread function because it 
corresponds to a very low, very wide skirt around the central blob. 

The size of the central blob is determined by the resolution of the 
device (or visa versa) and in our case has a half-intensity radius of 
around .09 mm (in the centre of the field of view). The scatter skirt 
however extends easily to the edge of the field of view 38 mm away. 

It is so low that it would go undetected due to dim-cutoff if we 
looking at point sources. Only when it is integrated over large areas 
is its effect noticable. It turns out that about 33 % of the incident 
light is scattered in this way. This causes a dramatic reduction of 
contrast. 

Several causes can be traced for this phenomenon. The lens contributes 
some small amount of scatter but the major defects occur because of 
multiple reflections in the face-plate and reflection from the aperture 
plate at the end of the drift-tube. It fs not known whether any electron 
optic effects come into this as well. The light enters the face plate 
and is partially absorbed by the photocathode; some light is,however, 
reflected and may bounce repeatedly inside the face-plate. Some fraction 
of the light also passes right through the photo-cathode and strikes the 
shiny nickel (I) arparture plate only to be reflected onto the back of 
the photo-cathode. 

These problems could be ameliorated by optically coating the face-plate 
to avoid multiple reflections or to use a fiber-optic front-plate. The 
arperture plate clearly ought to be made of some more reasonable material 
(to avoid the magnetic problems) and should be fairly non-reflective. 

(we might expect,by the way*that the front-plate scatter is worse for 
larger iris diameter (lower f-stops) because the light will be entering 
the face-plate from larger angles relative to the optical.axis) 
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As pointed out this phenomena only occurs when we are integrating 
signals over large areas. To measure the effect, then.,we have to 
illuminate large areas. One method involves the use of a series of 
white discs on a black background to be viewed by the image dissector. 
For each size disc one records the intensity at the centre. This 
method suffers from the fact that it is hard to find paper surfaces 
that have a high reflectivity (>50%) and others having a low 
reflectivity (O0%). The observed effect is then considerably lower 
than expected,* in addition, the scatter in the lens is included. 


A better method is that of removing the lens, using a point source of 
light (such as a distant lamp reflected in a metal sphere) and using the 
iris to allow variable diameter circles of light to fall on the 
photocathode. We observe in this way the integral of this scatter 
function. Let the point-spread function be rotationally symmetric, f(r). 



If we wish we can differentiate the observed function and get: 


2*iT f(r) r 


There is some reason to suppose that f(r) can be approximated by e -<rr /r 



h rv% 
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We can use our results to estimate the intensity at various points 
in an image consisting of large polygonal areas of uniform intensity. 



Let£ look at the intensity at the points A1, A2, B1, B2, Cl, C2 


assuming 33 % spill-over: 

A1 (90° out of 360° illuminated) 1. - .33 (3/4) = .75 

A2(” " " ” ” ) 0. + .33 (1/4) =.08 

B1 (180° out of 360° illuminated) 1. - .33 (1/2) = .84 

B2 ( " . ) 0. + .33 (1/2) = .18 

Cl (270° out of 360° illuminated) 1. - .33 (1/4) = .92 

C2 ( ” ... ) 0. + .33 (3/4) = .25 

D1 1. 

D2 0. 
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WHY THE VIRTUAL IMAGE OF A POINT-SOURCE LOOKS EQUALLY BRIGHT FROM 
ALL DIRECTIONS: 



Consider the small surface ring where the light is incident at 
an angle 0 w.r.t to the surface normal. 

The incident area is: 2*irr sin 0 cos 0 d0 = *trr 2 sin 20 dQ 

Light falling into this ring is reflected at an angle 20 w.r.t 

to the incident ray and with a spread 2 d0. At the distance R, 

the light reflected from the ring is spread into an area 
2 

2 R sin 20 2 d0. The intensity per unit area at distance R is: 

a _ 

I (it r sin 20 d0)/(2irR 2 sin 20 2 dfr) = I (r/R) 2 / 4 

So its independent of what angle one is looking at it from. 

This has implications for reflectivity models of surfaces made of 

spherical particles. It is also useful in producing point-sources 

with very small source areas (we are assuming both source and observer 

distant from the sphere). Note that the factor of 4 comes from the 

2 

fact that the incident islTr , while the light is reflected into 

2 

and area 4 *tTR . 






GLOB-TRACKING: 


Suppose we have an intensity glob such as a ping-pong ball against 
a dark background. The object is to track it using the random access 
camera. Define a two-dimensional pattern of points. The spread and 
position of this pattern will be servoed using the intensities read. 


At each step input the intensities, find their maximum and minimum, 
IMAX and IMIN. If IMAX is too small, go into search mode, otherwise 
calculate the following sums 


51 X. I 
1 1 


T Y. I. 
1 1 


7L i 


Then adjust the position: 


X n + 1 ' X n + < 


21 *j Ij 

Zi, 


- X n > 


n+1 


- Z. Y• I. 

Y + *( 1 1 

n v » v 


n 


- Y n ) 


i 


Then adjust the size of the pattern: 

IMAX'-IMIN’ 

A...* V l +^(-l) ) 


H-M 


IMAX -IMIN 


Where (IMAX 1 - IMIN') is the desired state of intensity range. 


Usually 


0 , > 9 1 


eg =1/2 


6 ^ 1/8 
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A RELATED ANALOG TYPE GLOB-TRACKER 



Here g(t) is some test function like cos (wt), for example, and f is the 
external function such as intensity. An interesting case is obtained if 
we combine two of these circuits, one for x and one for y coordinates in 
an image dissector camera. We then have a star-tracker. The two g(t)'s 
will need to be "orthogonal" then, cos(wt) and sin(wt), for example. 

A similar circuit or equivalent program can be used for light-pen tracking 
Interesting variations concern the question of whether the low pass filter 
can be eliminated or replaced by some other device and whether g(t) can be 
removed or "self-generated". In other words one aims at a system that is 
self-contained and samples the image in a way dependent on what is in the 
image rather than some fixed predetermined pattern. 
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THE SURVEYORS MARK AND FRIENDS: 

To track an object using the image dissector camera it is desirable 
to have to read the intensity at as few steps as possible at each time 
interval. The pattern to look at must also be designed for three conflicting 
requirements: ease of acquisition, ease of tracking in fast motion and 
accuracy of locating when stationary. The first two cause the object to 
be fairly large, the last requires that some point on it be well defined. 

The program should have no difficulty in processing the intensities 
read and should be fairly independent of distance and orientation of 
the pattern. A radially symmetric pattern with black and white areas 
seems suitable. In particular,one consisting of a number of intersecting 
lines with alternate segments filled in black and white seems a winner. 

The one used by surveyors uses two lines, our robotics calibration 
programs use three-line patterns. 



The image processing is simple. One reads the intensity at a number of 
points on the circumference of a circle, finds the maximum and minimum 
and sets up hysteresis thresholds. The lines are detected at the points 
where the intensity crosses both thresholds in sequence. The six points 
define three lines. The centre is then estimated to be near the point of 
minimum sum of squares of perpendicular distances to the three lines. 
Image motion between succesive scans can be almost the radius of the 
pattern, while its centre can be located extremely accurately by shrinking 
the sampling circle. 
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OBJECT ROTATION MATRIX: 

Consider an object rotated first about the x-axis (pitch, p), then 
about the y-axis (yaw, y) and finally about the z-axis (roll, r). 
We are interested in the corresponding transformation matrix: 


cos r cos y (cos r sin y sin p - sin r cos p) (cos r sin y cos p + sin r sin p\ 
sin r cos y (sin r sin y sin p + cos r cos p) (sin r sin y cos d - cos r sin p) 
“ sin y cos y sin p cos y cos o 



Left eye: x' = (x+s)f/z y* = (y)f/z 

Projection of point (x,y,z) 

Right eye: x" = (x-s)f/z y" = (y)f/z 

f is the distance the resulting images are to be viewed from. 

2s is the eye separation. 
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4 


EXPOSURE GUIDE FOR OUR DEC 340 DISPLAY 


I 0 + 1) 


3/2 


f = 



( 1 + 1 ) 


3/2 


f 

k 


s 

A 


N 

P 


0 

1 

2 

3 

4 

5 

6 
7 


1 

2.8 

5 

8 

n 

15 

18 

23 


f-number indicated on lens, 
empirically found to be about I /lS (gives rise to density of 
about 2in negative; i.e. almost overexposed). 

Half-intensity radius of spot on DEC 340, varies somewhat with I. 
use .5 mm unless you have good reason to suspect other value. 

Half-intensity radius of blur in camera projected back onto 
display surface - varies with lens and film used. - 

use .5 mm unless you have good reason to suspect other value. 
Spacing of points in image you are displaying. Use CO if all the 
points can be resolved in the image. 

S I 

Use .25 mm * 2 for vectors, increments anq characters of scale s. 
scale send to DEC 340, 0-3. 

ASA rating of film. 

For polaroid B/W: 3000 
For 35mm TRI-X : 300 

For 16mm TRI-X : 200 

Argument to .NDIS ; i.e. number of times points are displayed. 
Packing factor. 

1 ■ for resolved points. 


max(1, ' St 1 for one-dimensional sets of points (vectors,increments 

r 3 characters) 

r +r„ 2 

max(l,(J«i*) ) for two-dimensional sets of points (rasters). 

r 3 

F - Filter factor. 1 for no filter, 2 for Wratten 15 (afterglow only), 

8 for Wratten 47 (flash only). - 

I - Intensity parameter send to.scope. If varying intensities are to 

be recorded, use 1=5 - highlights will be slightly overexposed but* .... 
the dark-areas will not be completely under-exposed. 0-7, 
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SOME LENS FORMULAE: 



Let P 1 be the front principal plane, be ? 2 the rear principal plane. 

Let f be the focal length and the media on the two sides of the lens be 

the same. Let f-j be the object-lens distance and f« the lens-image 
distance. 



The de-magnification of the image is then: 


M = 


We know that: 


1 1 

«■» + m 

f l f 


1 

f 


i«e. - f)(f - f) = f‘ 


fl = (1 + M)f 
f 2 = (1 + 1/M)f 


Let d be the object to image distance (ignoring thick lens effect): 

2 


d = f l + f 2 = f (M + 2 + 1/M) = f 


(1+M) 


M 


• Let 


x = (-1) 

2f 


then 


M - 2,x M + 1 = 0 


x = tL-Li = -(M + 1/M) 

2M 2 


M = (x-1) + J x -1 
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dd 1 M 2 - 1 df, 

dM M 2 dM 




These formulae' are useful for calculating focusing accuracy for example. 

Numerical arperture is defined as ft sin(0/2), the f-stop as 1 

2 sin(9/2) 

Where £ is the angle subtended by the lens at the centre of the image. 

The intensity at the image is proportional to l/(f-stop) 2 . 



r t (Radius of curvature) 


Then we have the lens-makers equation: 



Optimal pin-hole radius (for diffraction to equal hole spread): 


r = 


■ 


where d is the hole-image distance, Xthe wavelength 


Airy radius: 


2 (3.8317/ (2*rr )) X (f-stop) = 1.22 X (f-stop) 


(Since 3.8317 is the first zero of J,(x)/x ) 
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FOCAL LENGTH CALIBRATION: 

For accurate camera models one needs good measurements of the focal 
length and the position of the principal, planes. 



We measure several combinations of x. and y. and attempt to find 
a, b and most important, f. We can assume that a and b are relatively 
small relative to f and that f is known approximately. We clearly 
require three such sets of measurements and could use least-squares 
methods if we had more. Unfortunately the equations are non-linear. 

We can make them into polynomials in a, b and f however: 

l/(x r a) + l/(y.-b) = 1/f 

((x^y^) - (a+b)) f - (Xj-aHy^-b) = 0 

-(ab+bf+fa) + (x^f+b) + y^f+a)) -x.y i = 0 

We could solve this set of second order polynomials in 3 variables 
in a number of ways. Perhaps the easiest is multi-dimensional Newton- 
Raphson iteration. We consider this last expression as a function F 
of the parameters a, b and f and are aiming for F(a,b,f) = 0. For this 
we require the derivatives: 

dF/da = y.j - b - f, dF/db = x.. - a - f, dF/df = (x^+y^ )-(a+b) 

We can also use guessing or a least squares method. If we can select 
the x. and y. we can also simplify the problem. 
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Special Case: If we can choose x.. and y. we might try the following: 


= tsO 

x, = f+a 

a = x-j-f 

= &o 

y 2 = f+b 

b = y 2 -f 


We need one more measurement: 


l/(x 3 - x 1 +f) + l/(y 3 - y 2 +f) = 1/f 


Let x 3 -x ] = x', y 3 -y 2 = y' 


(y'+f + x'+f) f - (x 1 + f)(y 1 + f) = 0 
(x'+y')f + 2f^ - x'y' -(x 1 +y')f -f^= 0 

-x'y' + f 2 - 0 f x'y' ' 

f =7 <*3- x i>(y 3 -y 2 )' 


For accuracy we want both differences large, this implies that 
we want x 3 about the same magnitude as y 3 . 
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DETERMINING THE TRANSFORM FROM ARM TO EYE SPACE: 
Being a rotation and translation we expect: 


x v 


a n 

a 12 

a 13 


x a 


a 14 

•Yy 

- 

a 21 

a 22 

a 23 


y a 

-4~ 

a 24 

2 v 


a 31 

a 32 

a 33 


z a 


a 34 


And the matrix ought to be orthogonal (i.e. A* A = I ). The coordinates 
with a-subscripts are arm coordinates, those with a v-subscriDt are 
eye coordinates. By allowing the matrix to be non-orthogonal we can 
absorb some of the distortions and non-linearities. In any case 
forcing it to be orthogonal introduces a non-linear constraint that 
messes up the mathematics 1 We then have to use iterative methods 
Well-known in the art of reducing aerial photographs. 

Next we have to consider the projection into the image plane: 

u = (x y /z v W + u Q 

v = (y v /z y )fi + v Q 

06 and (b are normally the same more or less and depend on the focal length 

and the translation from image coordinates to deflection units, u and v 

o o 

are zero if we choose the image origin on the optical axis which may at 

times be convenient. It is not hard to show that <*,/$, u and v can be 

< o o 

absorbed into our first transform and we can consider the simpler case: 

u = x /z and v = y /z 

V V J V / V 

This does make the matrix non-orthogonal however. Clearly, multiplying 
all the a..'s by any factor causes no change in the image coordinates 
and we can therefore choose a fixed value for one of them, say = 1. 

We then have the problem of determining the values of the other 11 terms. 



: - 
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We need at least 11 equationsthen andpreferably more so as to allow 
a least-squares solution. One method of determining the transformation 
matrix depends on moving the arm into n known positions and recording the 
corresponding x ., y ., z . and image coordinates u. and v.. It is 

a I a I ai j 1 

convenient to track a special object held in the hand as it moves around 
rather than to blindly move the hand and try and locate it in the image. 

For each such measurement we get 2 equations: 



x - z u. = 0 

V V 1 


and 


y - z v. = 0 
J'v v i 


a ll x ai +a 12 y ai +a 13 z ai +a 14 



For n such measurements we get 2n such equations which can be separated into 
two groups and written in matrix form as follows: 


x al 

y al 

z al 

1 

0 

0 

0 

0 

~ u l x al 

- u l y al 

-u,z , 

1 al 

- u i 

x a2 

y a2 

z a2 

1 

0 

0 

0 

0 

” u 2 x a2 

" u 2 y a2 

-U2; 

z a2 

~ u 2 

x a „ 

y 


1 

0 

0 

0 

0 

-U X 

-u y 

-u ; 

1 

-u 

an 

J an 

an 




n an 

n J an 

n 

an 

n 

0 

0 

0 

0 

x al 

y al 

z al 

1 

" v l x al 

" v l y al 

" v l z al 

- v i 

0 

0 

0 

0 

x a2 

y a2 

z a2 

1 

" v 2 x a2 

~ v 2 y a2 

-v 2 : 

! a2 

" V 2 

0 

0 

0 

0 

x,„ 

y 


1 

-V X 

-v y 

-v ; 

7 

~v 





an 

■'an 

an 


n an 

n J an 

n 

"an 

n 



a n 

1 °l 


a 12 

° 



* 


a 34 

1 ° J 
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Setting a 34 to 1 and taking the resulting constant terms (u ] ... u , v, ... 
to the right hand side we obtain 2n equations in 11 unknowns. We can make 
do with 5 1/2 experimental measurements or attempt a least-squares solution 
for n^. 6 points. Not more than 3 points should be in any one plane in the 
first instance to avoid degeneracy. It is convenient to use the points at 
the tips of an octahedron. 

Notes: 1. A slightly different formulation leads to 18 equations in 18 

unknowns’, the same results are obtained. (This corresponds to 
the homogeneous representation.) 

2. If we had assumed orthogonality we would have introduced 3 
more constraints and needed only 8 equations, that is 4 
experimental points which could conveniently be the corners 
of a tetrahedron. 



Page 303 


RELATION BETWEEN THE SIMPLIFIED AND THE REAL IMAGE COORDINATES: 


Given: 

> 

X 


a n 

a l2 

a l 3 


x a 


a 14 


y v 

— 

a 21 

a 22 

a 23 


y a 


a 24 


z v 


a 31 

a 32 

a 33 


z a 


a 34 


and 


u = V z v and 



v 


Where x g , y g , are coordinates relative to the arm coordinate system. 
x v , y y j z v are coordinates relative to the eye and u and v are image 
coordinates in the simplified system. 


Now we introduce the real image coordinates: 


u' = u + u 


o 


and v' = v A + v 


o 


x v ' t“'- u 0 ) z v = 0 and /*y v - (v'-v ) * 0 


C ° <a ll + V31 )x a +(o<a 12 +u o a 32^a +<0<a 13 +u o a 33 ) V (o<a 14 +u o a 34 ) 

• u ' (a 3lV a 32V a 33 z a +a 34) =0 

( / ia 21 +v o a 31 )x a +( / J a 22 +v o a 32> V ( / a 23 +v o a 33> V</* a 24 +u o a 34> 


- v ' (a 31 x a +a 32V a 33 z a +a 34 )= ° 


So finally: 


. 


X* 


V 


y' 


z* 


V 



(Va ll +u o a 31> ' ,<a 12 + V32 ) <* a 13 +u o a 33 ) 

( A a 21 +v o a 31 1 </ la 22 +v 0 a 32> (/* a 23 +v o a 33> 


31 


) ( 


32 


) ( 


33 


) 


( * a 14 +u o a 34 } 

( /* a 24 +v o a 34 } 


( a 


34 


) 
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A VERTICAL PREDICATE FOR IMAGE LINES: 

In a perspective transformation of the world a set of parallel lines 
will project into a bundle of lines passing through one point, the 
vanishing point x,, y, . 

means parallel to the arm's z-axis: 

y v a 23 z a ’ z v -* a 33 z a 
and y f = a 23 /a 33 

In practice vertical means perpendicular to the table. Suppose the table 
equation is given by: 


If we simply assume that vertical 
Then as « , x a 13 z , 
And so x f = a, 3 /a 


P-|X + P 2 y + P 3 z + P 4 = 0 

Then let x a = *>< p-| , y a = <*P 2 » z a = ^3 and let <30 . 

x v * *( a ll p l + a 12 p 2 + a l3 P 3^ 

y v * * ^ a 21 P 1 + a 22 p 2 + a 23 P 3^ 

z v * * ^ a 31 p l + a 32°2 + a 33 p 3 ^ 


x = a ll p l + a 12 p 2 
a 31 p l + a 32°2 
_ a 21 p l + a 22 p 2 

y f - 

a 31 p l a 32°2 


a 13 p 3 


+ a 33 n 3 
+ a 23 P 3 


+ a 33°3 


To test if a line is vertical or near vertical we calculate the angle it 
makes with the line connecting it to the vanishing point: 


x 12 X 1 " x 2 * x lf X 1 " x f * y 12 = y l ” y 2 * y lf = y - ] ~ y 
Csin ©) 2 = (x lf y ]2 - x 12 y ]f ) 2 / (x 2 f + y 2 f )(x 2 2 + y 2 2 ) 
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GOING FROM IMAGE COORDINATES TO ARM SPACE COORDINATES: 

Clearly we need some extra information to make up for the lack of 
one dimension. But first lets look at what we have: 

u = x y /z v and v = y v /z v 

x y - u z y = 0 and y y - v z = 0 

C a 11 -ua 31 )x a +(ai 2 - ua 3 2 )y a +(a 1 3-u a 33)z a =(a 14 - u a 3 4) 

( a 2r va 31 >x a +(a 22' va 32^ a+ta 23- va 33 )z a =(a 24- va 34> 

We need a third equation in x,» y 3 and z, to be able to solve. We could, 

a a a 7 

for example,be given any one of these three coordinates. More likely is 
the case where we have some relation to the table. Let the equation of 
the table be given by: 


P]X + P 2 y + p 3 z + p 4 = 0 

If the point is on the table we can simply use this equation. 

If the point is in the same plane parallel to the table as some other 
known point x-j, y-j, z, then we use the equation: 

p lV p 2 y a + p 3 z a = p l x l + p 2 y l + P 3 Z 1 


If the point is directly above (along a line normal to the table) some 
other known point y 2 > z 2 Then we have: 


<V X 2> P3 


- (V z 2> h ■ 0 
(y a -y 2 ) p 3 - (v z 2> H * 0 


We only need one of these and will use the first since it comes out more 
accurately with the eye-arm geometries we use. 
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x u 


'0 1 o 


X 


0 

V 




a 



y v 


-sin 6 0 cos 6 


y a 


sin 0 x„ 

0 

z w 


-cos 0 0 -sin 0 




cos 0 x + ? 

V 




a 


o 


Now we change to real image coordinates: 


X 1 

x v 


(tx? a ii +u 0 a 3 i) (^a 12 +u 0 a 32 ) 



^ a 14 +u o a 34^ 

y; 


(/^ a l 2 +v o a3 l) ^ a 22 +v o a 32^ ^ a 23 +v o a 33^ 

y a 

+ 

^ a 24 +v o a 34^ 

z ; 


( a 31 ^ ^ a 32 ^ ^ a 33 ^ 

z a 


( a 3 4 ) 


So we get: 


*; 


-u cos 0 

*< 

-u sin 6 

0 i 

; 

x a 


u o (cos 0*x o +fj 

y v 

r 

- As in 0 - v Q cos 0 

0 

CD 

to 

a 

> 

1 

CD 

CO 

O 

o 


y a 

+ 

/£in 0 x 0 +v Q (cos 0 x Q +t) 

z ; 


-cos 0 

0 

-sin 0 


z a 


cos 0 Xq+£ 
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Now suppose 6-= 30°, cos 8 = .86... , sin 6 = .5, u Q =v =512. (Center of 
coordinates for the image dissector on a scale of 0 - 1024.) 

With a lens of 10" focal length we find that <* =3160 units/radian. 

With a lens of 6.5 " focal length e* =2000 units/radian. 

(Assuming about 12.5 units per mm on the photocathode) 

Next suppose x Q = 30.0", f = 50.0" and use of the 6.5" lens. 


-440. 

2000. 

-256, 


39800. 

-1440. 

0 . 

1464. 


69800. 

-.86 

0 . 

-.5 


76. 


Next we normalise by setting a 34 = 1: 


-5.8 

26.3 

-3.38 


525. 

-19.0 

0 . 

19.3 


918. 

-.0113 

0 . 

-.0066 


1 . 
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ABSOLUTE ORIENTATION: 



X D 


t* X„ 


X 

P 


P 


0 

v p 

= R 

^ y p 

+ 


Z P 


X z p 


z o 


Where R is an orthogonal rotation matrix. are scale factors, 

often equal to one another. x Q , y , z Q is a displacement vector. There is 
one of these equations for each point in the object. 


Let X • • ™ X • and x • • — x • “ x • > then: 

* J * J * J * J 


x ij 


* x i j 

v 1j 

= R 

^ y ij 

7 

/r, • ■ 

1J 


* z ij 


Multiplying this equation by its transpose we get: 



x ij 



* x ij 

X. . Y. . Zijl 
1J 1J i 

Y u 

= r* cx i j 

Mj & z ijl rTr 

1 



Z ij 


J 

|^iJ 


Now noting that R T R = I we get: 
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Now suppose we are given four points in each coordinate system: 


< X 12 + V 12 + Z 12> 


2 2 2 
x 12 y 12 z 12 



<*23 + V 23 + Z 23> 

s 

2 2 2 
x 23 y 23 z 23 



<4 + 4 + 4 > 


2 2 2 
x 34 y 34 z 34 




It is now easy to solve for «, /l,X . Let x! . =rtx.. and so on: 

* vJ * vJ 


X 12 


x 12 

y 12 

7 * 
z 12 


r n 

X 23 

* 

X 23 

y 23 

7 1 

z 23 


r 12 

X 34 


X 34 

y 34 

7 1 

z 34 


r 13 


Let X be the vector [x ]£ X 23 X 34 '] T and similarly for Y and Z. 

Let x be the vector £ x j£ x 33 ^ 34 !^ and similarly for y 1 and z 1 . 

Combining three equations like the above: 

(X Y Z) = (x* y' z') R T 

R T = (x' y' z 1 )" 1 (X Y Z) 



X 12 y 12 z 12 

- 1 

X 12 Y 12 Z 12 

£ 

x 23 y 23 z 23 


X 23 Y 23 Z 23 


x 34 y 34 z 34 


X 34 Y 34 Z 34 


Due to measurement inaccuracies R determined this way may not be orthogonal, 
one can if one wishes adjust it iteratively using Newton-Raphson: 
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R 


n+1 


R 


n 


• 5 ( Rj R. - I ) 


or 


R 


n+1 


■ - 5 < < R l > _1 + R n > 


Finally we have to find the displacement vector: 


x o 


X P 


* x p 

*0 

— 

y p 

- R 

/* y p 

z o 


Z P 


Vz p 


We can get four estimates from this which we can average if necessary. 
If « = we can get away with only three points. 


GEOMETRY OF THE AMF-VERSTRAN ARM WITH THE ALLES HAND: 


(ROLL) & 


LO 

2.75" 

LO.5 

1 .0" 

LI 

3.625 

LI.5 

1 .0" 

L2 

10.5" 

L2.5 

.75" 

L3 

4.75" 

L4 

6.375 

L5 

.25" 

L6 

2 .0" 

L7 

2.75" 

L8 

.56" 


LO.5 


(YAW) 


LI 



db 


"VERTICAL" 


(a, 

e<u 


SWING 


LO 


t 


"HORIZONTAL" 


LI.5 


L2 


L3 



ROTATE 


■4T3 


ytr^iu l 

L2.5 


a 


TILT 


EXTEND 


L6 


L7 



vide 


4-GRIP 


L4 
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COMPENSATION FOR GRIP MOTION: 



The geometry of the grippers is equivalent to the above. We then find 
a motion along the axis of the grippers w.r.t. the most extended: 



COMPENSATION FOR TILT MOTION: 

When the tilt-axis is inclined 0 w.r.t vertical one can adjust the 
horizontal extend by sin 0 * hand-extend and the vertical motion by 
cos 0 * hand-extend. 

On the whole, the arm geometry is very simple and allows direct 
determination of joint angles and extensions given a desired hand 

position and orientation, keeping in mind that one only has 5 degrees 
of freedom. 
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CONVERSION FROM RECTANGULAR COORDINATES TO PSEUDO-POLAR: 

The AMF arm has an offset in its otherwise extremely simple geometry: 



Given x, y we need to find R and <*„. 

R = Jx 2 + y 2 - r 2 ' x 2 + y 2 > r 2 


tan ot, = x/y 


tan <*„ = 1/ tan (<*,- 0 ^) = 


yR + xr 
xR - yr 


tano^ = r/R 
xy + rR 

~ . .a 

)C - 


xy + rR 

IFT7 


Here we cannot avoid the use of arctan because we actually need the angle. 
Note that for the AMF arm r = 2.75" . 

We used the fact that x 2 - r 2 = R 2 - y 2 

And that tan(a-b) = (tan a - tan b) / (1 + tan a tan b) 

R is the "horizontal" extend and <* 0 is the "swing". 
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MAINTAINING A CONSTANT HAND ORIENTATION: 


tilt 


3 RoTftTe 


1 ? f , 


©t SvifjC 


Let the normal unit vector to the plane containing the two fingers be (a,b,c) 


/cos*/ sin 0 
(a b c)=(-sintt* coso< 0 


cos0 0 sin®\ A 0 0 

0 10 If 0 cos^ sin 

sin£ 0 cos$/ \p -sin^fc cos jh 


cossin 6 cos <f) + sin sin y 
= -sin«< sin 0 cos <p + cos^ sin 
cos £ cos M 

if. ... / 

• ■* ■ ■ ■ K 

* ‘ ' t * 

J . , ' . ' . • ;• 

So given one constraint we can solve foro<j0 } ^ 


2 2 2 

keeping in mind that a +b +c =1 


Most commonly we would be given the swing,^ , then 
sin^ =(a sinot + b cos ) 


tano =(a cos - b sin^C )/c 
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MEASURING THE INERTIA OF A LINK IN AN ARM: 


¥ 

For fast arm motions one requires a good dynamic model of the arm, 
including the geometry of joints and motor torques. Also required is 
the approximate moment of inertia of the links in the arm. It is 
usually not feasible to calculate these because of the complex shape 
and number of parts a link is made of. A simple empirical method 
requires one to'mqasure the total mass and distance of the centre of 

'■A . , ... .■ •%.•«* . . " . 

gravity from 1 the connection to the preceeding link as well as the 

< - , • *# *> 'k. v 


period of osci 


‘ * 

ion 




is suspended from this connection. 



* < , 
’ * * 

% 

v> 


Let 


= mass of link, 1 = distance of c.g. from preceeding connection 
= acceleration due to gravity (9.8 meter/second 2 ) :: " 

= period of oscillation, I = moment of inertia 


I *0* = -mgl sin 0 


9 = -(mgl/1) 0 


0 = A cos(,/mgl/l )t = A cos (2rrt/T) 
T = ZrrJ I/(mgl)' 

I » mgl (T/2ir) 2 


Example: Pendulum made of string and heavy weight: T = 2 TT/(1/g) 

I = mgl (1/g) = ml 2 











